Blameless Incidents: Learning from Failure at Scale
summary
Facebook runs a blameless incident program that trains every engineer in week two of bootcamp, classifies outages as SEV 0–4 (0 worst), and uses a central Site Event Manager to log timelines, notify experts, and open incidents company‑wide. Incident managers (eyemocks) coordinate responses for 40,000 staff across 2.4 billion monthly users; routine playbooks use Google Docs/Quip, major “black swan” failures trigger forensic reviews and six‑month re‑reviews. No punishment for reporting; followups fix root causes (example: compiler change after a bootcamp‑triggered outage).
AI-written summary · about
chapters (19)
- 0:00Intro
- 1:02Quick Show of Hands
- 1:45Process Goals
- 3:00Routine Outages
- 3:54Unpredictable Events
- 4:51Black Swan Failures
- 6:09Severity Levels
- 8:46Scale Brings Complexity
- 9:48Culture: Teaching and Learning
- 11:37Early Stages of Incident Management
- 12:34Scaling Incident Management
- 13:45Scaling Limits
- 16:25Notable Evolutionary Steps
- 18:14Facebook IM Today
- 19:41Main SEV Review
- 23:49Incidents Can Hurt
- 25:53It's about Teamwork
- 27:36Some Unsolicited Advice
- 29:29A Real Outage from Years Ago
related talks
37mIncidents are a new normal
27mManaging Incidents
30mLearning From Failure: Post Mortems
20mHow to Have an Operational Incident
34mIncidents are new normal
57mThe Curious Incident in your Software in the Day Time
22mBlame, shame and panic - how not to respond when things go wrong
19mHow to handle incidents at scale