The Art of Turning Incidents into Opportunities
summary
An SRE at Datadog describes accidentally upgrading a Mesos cluster that breaks scheduling, a production outage fixed in about one to two hours, and a response pattern that prioritizes quick mitigations, clear roles (incident commander), and coordination via testable tools and runbooks. Concrete practices: declare test incidents, run playful drills like capturing the Stray Pokemon to rehearse coordination, read outage reports weekly, and “ditch the template” in favor of narrative reports that capture frustrations, near-misses and action items.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 3:13First production break
- 8:51Fear of incidents
- 11:40Preparing for incidents
- 16:37Incident response
- 23:00Post-incident reflection
- 27:15Incident learning culture
- 30:43Embracing surprises
related talks
15mSustainable Incident Management for happy SRE teams
30mHow Did Things Go Right? Learning More from Incidents
15mIncident Ready: Chaos Engineer Your Incident Response
22mSRE and Risk management: a love story
13mNavigating SRE/Incident Management
39mHuman Observability of Incident Response
45mFailure Happens: Improving Incident Response in Large-Scale Organizations
28mRevolutionizing Incident Response