After the dust has settled: learning from incidents
summary
After an incident ends the actual work is concrete: recreate missing servers after an authorized deletion wiped a whole internal cluster with no backups, rebuild service state from deleted machine configs, and recover from Facebook’s Oct 4 2021 outage where a routine backbone check accidentally takes down the entire backbone and subsequent systems delete DNS records so global reachability disappears. Remedies used include screenshots from dashboards, 72 hours or five days deadlines, a designated follow‑up owner, a postmortem reading group, and exercises like wheel of Misfortune and chaos runs to validate fixes.
AI-written summary · about
chapters (7)
- 0:00Introduction and personal anecdotes
- 2:52Incident life cycle and retrospectives
- 7:24First incident and lessons learned
- 14:04Learning from incidents and the Learning Pyramid
- 17:35Facebook outage and DNS insights
- 22:31Recap and learning from incidents
- 24:35Q&A on retrospectives and accountability
related talks
30mLearning From Failure: Post Mortems
15mIncident Ready: Chaos Engineer Your Incident Response
30mLearning from Learnings: Anatomy of Three Incidents
34m"The Upside of Downtime: How to Turn a Disaster Into an Opportunity"
22mBlame, shame and panic - how not to respond when things go wrong
28mPost-mortems or How to Walk Away Happy from a Carnage
19mStorytelling as an Incident Management Skill
44mThe role of catastrophic failure in software design