How to handle incidents at scale
summary
Defines incidents as unexpected disruptions and prescribes a concrete workflow: detection via tools like Firebase Crashlytics, Datadog and Prometheus with thresholds (example 98% API success threshold), alerting into a dedicated incident channel, then an incident response team split into debuggers and an incident communicator role who gives 15–30 minute updates. Resolution prefers rollback over risky hot fixes when possible, followed by a required postmortem (examples: GitLab DB outage, Atlassian template) before declaring closure.
AI-written summary · about
chapters (12)
- 0:00Introduction and background
- 1:07Defining incidents and benefits
- 2:34Three phases of incident handling
- 3:55Detection: classification and monitoring
- 6:22Detection: alerting and preparedness
- 7:25Incident response team structure
- 9:10Communication: roles and channels
- 10:54Communication: updates and transparency
- 12:58Customer communication and status pages
- 14:23Resolution: fixes and monitoring
- 16:06Postmortem: analysis and prevention
- 18:12Closing the incident
related talks
20mData Incidents - When things go wrong
28mHow to Have an Operational Incident (A Crash Course)
27mManaging Incidents
37mIncidents are a new normal
30mHow to have an operational incident a crash course
51mDatabase Incident Management: When Everything Goes Wrong
30mHow To Have An Operational Incident (a crash course)
11mUsing incidents to level-up your teams