Whispers in the Chaos: Monitoring Weak Signals
summary
Argues that failures become learning only when incident response and systems capture what actually happened: concrete examples include Knight Capital losing $480M (about $140k/sec) after a bad deploy, a pet-feeder outage that stopped animals being fed, and an S3 typo that removed ~30 servers and broke the service-health dashboard because it lived on S3. Recommends injecting faults (including chaos monkey and human/hardware drills), running timely debriefs, using chatops timelines, practicing incident command, and making infrastructure retrospective-ready.
AI-written summary · about
chapters (16)
- 0:00Start
- 2:18Get Some Paper!
- 3:38Thought Experiment: An Incident in the JFK "Datacenter"
- 10:43Accident Blame Success Retrospective
- 14:16Safety as "Energy & Barriers"
- 19:17Safety Impossible Due to "Normal Accidents"
- 22:21Safety in "High Reliability Orgs"
- 30:28Successful Failure: The Methods
- 34:54Debrief the Actors
- 35:12Gather the Data
- 35:28Create a Timeline
- 44:17An Antipattern: Stopping the Line is a "Privilege"
- 47:41A Bit About Biases
- 49:25A Sea Of Biases
- 50:25A Simple, Surprising Example
- 54:04Successful Failure: The Land Mines
related talks
01h 03mWhispers in the Chaos: Monitoring Weak Signals
41m‘Failure’ as Success: The Mindset, the Methods, and the Landmines
40mWhispers in Chaos: Searching for Weak Signals in Incidents
19mLearnings from Chaos experiments
45mDeveloping a Chaos Architecture Mindset
57mChaos Engineering Primer
15mIncident Ready: Chaos Engineer Your Incident Response
01h 02mPractical Chaos Engineering breaking things on purpose to make them more resilient against failure