Incident Groundhog Day
summary
A staged‑world exercise recreates a Black Friday e‑commerce outage triggered by a secret API key rotation plus a cache refresh; 20 incident responders each face the same scenario and 13/20 fix it within 25 minutes. Resolution time shows no clear link to seniority, and arguing over severity consumes repair time. Practical findings: therapeutic actions like rollbacks surface diagnostic signals (rollbacks become diagnostics), teams split into solo heroes or collaborative bands, and a tight update template keeps responders focused (severity arguments waste time).
AI-written summary · about
chapters (7)
- 0:00Groundhog Day setup
- 3:11Incident response challenges
- 5:25Simulating incidents with war games
- 10:44Latent conditions in incidents
- 13:32Analyzing incident response
- 25:43Simulation limitations
- 27:57Building resilience through practice
related talks
32mSurviving Black Friday - A resilience engineering tale
15mIncident Ready: Chaos Engineer Your Incident Response
11mUsing incidents to level-up your teams
22mDon’t wait for an outage to improve your reliability
45mFailure Happens: Improving Incident Response in Large-Scale Organizations
40mWhispers in Chaos: Searching for Weak Signals in Incidents
45mLet's play a real life Dungeon and Dragons
30mUsing incidents to level-up your teams