After the dust has settled: learning from incidents
summary
Post-incident work focuses on concrete actions: capture logs and Slack transcripts immediately, draft the postmortem within 5-day draft to preserve graphs, and track remediation to avoid regressing. Real incidents include a cleanup that deleted long-tail VMs because of Meltdown/Spectre and revealed missing disaster recovery, and a global networking failure that causes a 6-hour outage and locks engineers out of datacenters. The useful outputs are named ownership, exact rollback/playbook steps, and the one thing that actually fixed production.
AI-written summary · about
chapters (6)
- 0:00Introduction and Marvel references
- 3:01Incident lifecycle explained
- 7:18Personal incident story and lessons
- 13:20Learning theory and postmortems
- 16:13Learning from others' incidents
- 20:33Recap and responsibility
related talks
23mYou put out the fire on production, what’s next?
37mIncidents are a new normal
15mPragmatic Incident Response: Lessons Learned from Failures
30mLearning From Failure: Post Mortems
42mYou put out the fire on production, what's next?
34mIncidents are new normal
36mSurviving in Production: Incident Management and Learning
31mThe Art of Turning Incidents into Opportunities