Don’t wait for an outage to improve your reliability
summary
Claims that game days give the full reliability gains of real outages without customer impact and that they’re repeatable using concrete breakage tactics. The process uses DNS rule blocking of a third‑party, a malicious endpoint that calculates pi to spike CPU, and manual declaration when deployment must be simulated. Planning produces three headline scenarios, uses runbooks, a status page, scribe for call transcription, a private villain channel, and recommends 3–4 responders per 2–3 incidents in a half‑day session with follow‑ups that get fixed.
AI-written summary · about
chapters (10)
- 0:00The cost of incidents
- 1:17Learning from failures
- 4:40Introduction to game days
- 5:41Setting the stage
- 8:56Planning scenarios
- 12:23Logistics and team setup
- 14:15Game day execution
- 16:10Managing the simulation
- 19:05Wrap up and cleanup
- 20:12Reflecting and celebrating
related talks
15mMaintaining reliable systems
45mLet's play a real life Dungeon and Dragons
24mThe Formula for Faster Outage Recovery
52mPerformance & Stability Anti patterns
16mEngineering Failure-Resilient Systems
30mIncident Groundhog Day
42mPlan for Unplanned Work: Game Days with Chaos Engineering
39mManaging Misfortune for Best Results