Recognizing Zombies, Black Holes & Tribbles Before You Get Eaten
summary
A practical set of defenses: an open source work-tracker that uses a custom ticketed concurrent collection to stream in-flight requests, detect long-running “zombies” (use web-proxy five‑minute timeouts and interrupt or proxy-kill), identify black holes by computing a mean + 5σ threshold after ~50 samples to auto-open circuits and auto-heal, and throttle “tribbles” with concurrency limits returning 429 + Retry-After. Real incidents include a client fetching 1,700 items; the tools reduce JVM resource exhaustion and avoid manual resets.
AI-written summary · about
chapters (9)
- 0:00Introduction and motivation
- 10:00Observability and JVM tracking
- 13:39Recognizing and killing zombies
- 17:39Detecting black holes
- 23:43Black hole prevention demo
- 26:13Handling edge cases in detection
- 28:41Preventing tribbles
- 33:02Managing resource limits
- 36:15Open source projects and conclusion
related talks
32mDetecting Events on the Web with Java, Kafka and ZooKeeper
34mDistributed Systems - What Can Go Wrong and How to Get It Right
47mManaging and monitoring a JVM
23m30 Minutes To Zero: Slashing Debug Time With WhatIsWrongWithThis?
24mGremlins Exposed: Shining a Light on Mischievous Systems
29mResilient by Design
45mCatching the 137-Killer: A Java Memory Forensics Investigation
31mWhat Breaks Our Systems: A Taxonomy of Black Swans