‘Failure’ as Success: The Mindset, the Methods, and the Landmines
summary
Argues that success is not just lack of outages but how teams respond, and that the one controllable variable in complex systems is reaction. Cites Knight Capital losing about $460M in 45 minutes, a pet-feeder outage that left animals hungry, and Netflix’s Simian Army (chaos tools) as concrete examples. Recommends injecting failure via chaos monkey and human drills, keeping retrospective-ready infrastructure (black-box logs, chatops timelines), running timely postmortems within ~72 hours, and adopting blame-aware post-mortems to surface fixes.
AI-written summary · about
chapters (14)
- 0:00Intro
- 0:57Thought Experiment An Incident in the JFK "Datacenter"
- 4:03A Question
- 10:49Safety as "Energy & Barriers"
- 16:44Safety in High Reliability Orgs
- 23:18Successful Failure: The Methods
- 24:42Blameless?
- 25:34Debrief the Actors
- 26:03Create a Timeline
- 29:00An Antipattern: Stopping the Line is a "Privilege"
- 29:37Successful Failure: The Land Mines
- 31:43A Bit About Biases
- 36:18Back to Our Thought Experiment
- 39:23Takeaway #3
related talks
31m"Failure" as Success: The Mindset, the Methods, and the Landmines
29mLooking under the rug: the art of learning from failure
16mEngineering Failure-Resilient Systems
05mFailure and the Third Way
06mIgnite: Failure and the Third Way
01h 06mCongratulations, you Failed!
01h 03mWhispers in the Chaos: Monitoring Weak Signals
45mThe human nature of failure & resiliency