Learning from Firefighters to Improve Systems Reliability
summary
Adapts the Ten Standard Firefighting Orders (codified since 1957) into concrete SRE practices: keep a 120‑page pocket playbook with prerequisites, run 3–5 chaos tools from the CNCF landscape and regular game days to validate behavior, add honeypots and controlled burns to exhaust attack surface, and use a backstage plugin or simple documented wizards to pick the right playbook. Shares concrete failure stories (wildfire growth from 60 to 3,000+ acres, multimillion‑euro outage examples) and insists on declaring incidents early and keeping written timestamps.
AI-written summary · about
chapters (19)
- 0:00Introduction
- 1:06Speaker background
- 3:59Man Gulch fire lessons
- 7:31Firefighting orders overview
- 8:54Keep informed on system
- 9:58Consider weather factors
- 11:24Chaos testing retrospectives
- 13:10Graceful degradation design
- 14:35Communication in incidents
- 15:40Proactive honeypots
- 16:22Controlled burns fatigue
- 17:49Documentation training
- 19:35Declare incidents early
- 21:18Clear written orders
- 23:31Root cause analysis plural
- 25:36Protect system aggressively
- 28:51Command and control
- 31:04Reliability depends on team
- 34:04Q&A
related talks
15mSustainable Incident Management for happy SRE teams
32mSustainable Incident Management
21mWhy DevOps is Like Firefighting at Sea
44mDeveloping a Culture of Reliability and Resilience through Chaos
22mLooking Back to Look Forward
31mThe Art of Turning Incidents into Opportunities
45m“Disorganizing” Your SRE Organization
29mThe Curious SRE: Cultivating a Mindset for System Resilience