The story of firefighting: learnings from the incident management
summary
Smartwatch stress spikes (heart rate 177 vs normal ~70–80) match incident timelines, so incident work needs tooling and choreography: define three roles — coordinator, firefighter, communicator — and use PagerDuty or Grafana OnCall, Slack/Zoom, runbooks and a firefighting dashboard. Prioritize with P0/P1/P2 rules (checkout drop = P0, order-tracking view broken = P1, internal ETL job = P2), use SLOs, logs, traces and canaries, run postmortems (RFC/reason-for-outage) and simulate failures; AI for anomaly detection and postmortem drafting is emerging.
AI-written summary · about
chapters (12)
- 0:00Introduction and motivation
- 1:48Why incidents are inevitable
- 3:35Introducing Alex’s food delivery platform
- 6:09Building a firefighting foundation
- 7:54Prioritizing incidents by impact
- 11:51Observability and monitoring
- 14:39Firefighting strategy with Shisa Kanko
- 17:51What not to do during incidents
- 18:52Postmortem and continuous improvement
- 22:04Learning from incidents
- 22:45AI’s role in incident management
- 24:32Q&A and closing remarks
related talks
46mAlert, everything's on fire! How to manage technical incidents
01h 04mIncident Management - Talk the Talk, Walk the Walk
30mHow to have an operational incident a crash course
19mYou Can't Stop Fires with an Ambulance
39mNavigating the Chaos: A Holistic Approach to Incident Management
15mSustainable Incident Management for happy SRE teams
48mIncident Management - Talk the Talk, Walk the Walk
45mWhat to do when complex systems fail: A guide to chaos management