Whispers in the Chaos: Monitoring Weak Signals
summary
Argues that the valuable thing is not the topic but concrete practices: simian army chaos tools (including human and hardware drills), chat‑ops and teammate screen sharing for reconstructable timelines, and formal incident command plus debriefs. Gives disaster numbers — Night Capital loses $140k/second and $480M total — and cites S3’s “typed 30 not 3” partitioning/config checks failure. Recommends making systems retrospective ready, rotating crews (six‑week rotation), and empowering anyone to stop the line so responses improve and lessons actually stick.
AI-written summary · about
chapters (16)
- 0:00Start
- 2:18Get Some Paper!
- 3:38Thought Experiment: An Incident in the JFK "Datacenter"
- 10:43Accident Blame Success Retrospective
- 14:16Safety as "Energy & Barriers"
- 19:17Safety Impossible Due to "Normal Accidents"
- 22:21Safety in "High Reliability Orgs"
- 30:28Successful Failure: The Methods
- 34:54Debrief the Actors
- 35:12Gather the Data
- 35:28Create a Timeline
- 44:17An Antipattern: Stopping the Line is a "Privilege"
- 47:41A Bit About Biases
- 49:25A Sea Of Biases
- 50:25A Simple, Surprising Example
- 54:04Successful Failure: The Land Mines
related talks
01h 03mWhispers in the Chaos: Monitoring Weak Signals
23mHow to tame chaos effectively?
45mWhat to do when complex systems fail: A guide to chaos management
48mHarnessing Chaos; the hidden ingredient behind building better systems
15mIncident Ready: Chaos Engineer Your Incident Response
37mChaos Engineering: The Fine Art of Breaking Stuff in Production on Purpose
47mChaos Engineering Bottom Up
19mLearnings from Chaos experiments