Plan for Unplanned Work: Game Days with Chaos Engineering
summary
Explains how PagerDuty runs controlled chaos engineering as scheduled failure Fridays (now “failure any days”), injecting real-world variables like killed servers, severed networks or simulated region outages, measuring a defined steady state and using a hypothesis formula “under X circumstance…”. Experiments run with an incident commander, scribe and rollback plan, sometimes in 24 minutes, treated as real incidents, followed by post‑incident reviews and backloged fixes; incident commanders rotate every 48 hours.
AI-written summary · about
chapters (31)
- 0:00Introduction and conference opening
- 1:22Microservices dependencies and incident risks
- 4:14Netflix Chaos Monkey and Chaos Kong history
- 5:58What is chaos engineering and its principles
- 7:46Four principles of chaos engineering
- 8:28Steady state and system monitoring
- 9:32Formulating chaos engineering hypotheses
- 11:17Real-world variables in chaos experiments
- 11:58Minimizing blast radius in production
- 13:00Game days and incident response practice
- 14:27Testing observability and escalation policies
- 15:30Incident Command System and team roles
- 17:17Duration and post-incident reviews of game days
- 18:00Verifying steady state and incident response
- 19:24Defining groundwork for experimentation
- 20:06Testing alerting and notification systems
- 21:29Troubleshooting and resolution practices
- 22:33Post-incident reviews and knowledge sharing
- 23:58Avoiding tribal knowledge and sharing findings
- 25:00Revisiting SLOs and error budgets
- 26:02Surprise game days and organizational readiness
- 27:25When to and when not to run game days
- 29:49Reorganizations and timing considerations
- 31:14Leadership buy-in and organizational support
- 32:40Benefits of chaos engineering and game days
- 35:28Team onboarding and incident response training
- 36:31Planning game days at PagerDuty
- 38:58Post-incident review and documentation
- 39:39Starting small and scaling chaos engineering
- 40:43Key elements of a chaos experiment hypothesis
- 41:26Summary and closing remarks
related talks
47mChaos Engineering Bottom Up
45mWhat to do when complex systems fail: A guide to chaos management
03mChaos Day: When Reliability Reigns
26mPsychology of Chaos Engineering
27mChaos Engineering: The Path to Reliability
46mEmbracing Collaborative Chaos: Chaos Days on Large Platforms
29mThe Psychology of Chaos Engineering
29mPractical distributed systems: Building for the real world with chaos engineering