Planning for and handling failures from open hardware, aviation, to production at Google
summary
Covers concrete failures and fixes: LiPo batteries that reignite days later and using protection boards, the Samsung Note7 root cause of slightly thicker cells shorting, how mkdir -p 0755 /path/to/dir created a root-owned directory and took services down, and MCAS design choices that led to two 737 Max crashes. Recommends specific practices used in production: revert first, blameless postmortems, unit tests and CI, percent rollouts, rate limits on destructive ops, emergency “big red” rollback, and scripted disaster drills to validate runbooks.
AI-written summary · about
chapters (34)
- 0:00Intro
- 3:15Managing failures
- 4:58Eusebio
- 7:12Be mindful
- 8:00Hardware
- 10:29Phone
- 12:48Spare to spare
- 15:48Software
- 16:55Code Reviews
- 18:03Change Requests
- 20:06Unit tests
- 20:44Continuous integration
- 22:00File updates
- 23:00Postmortems
- 24:09Practicing emergencies
- 25:19Have backups be careful
- 26:57Disk Erase
- 28:02Rate Limits
- 28:36Postmortem
- 29:21Personal Lessons
- 30:19Aviation Lessons
- 32:04Risk Management
- 32:41Post Mortem
- 33:10Automation
- 33:58Selfdriving cars
- 34:47Air France 447
- 37:18Airbus QF32
- 37:59Indonesia
- 38:36Aircraft accident
- 39:25Boeing
- 43:41Certification
- 44:54Make a difference
- 45:27Conclusions
- 46:03QA
related talks
50mLearning in Production (or Why the Apollo 11 Landing Nearly Failed)
44mThe role of catastrophic failure in software design
30mLearning From Failure: Post Mortems
01h 06mCongratulations, you Failed!
26mThe top IT failures in history
41m‘Failure’ as Success: The Mindset, the Methods, and the Landmines
45mThe human nature of failure & resiliency
50mDevOps Project Failures - Lessons Learned and Strategies for Recovery