Breaking Production to Sleep Better: How and Why We Destroy Our Systems on Purpose
summary
Engineers at OVHcloud describe intentionally breaking production systems via organized "Game Days" to validate recovery procedures and uncover real failure points. They explain how to design realistic failure scenarios and ramp up from manual sabotage to large-scale load tests, illustrating with a stress test of a Grafana Mimir cluster (deployed on VMs), which revealed failures in network bandwidth, disk I/O, and Linux kernel limits rather than the application itself.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 3:07Why resilience proof
- 6:15From experiment to culture
- 11:49Game Day approach
- 17:32Cassis load testing
- 19:28Stress test failures
- 22:59Overlooked dependencies
- 26:56Stopping conditions
- 30:14Key findings
- 33:15Applicability beyond
- 36:35Chaos vs benchmarking
- 41:08Communication inspiration
- 41:55Smaller chaos experiments
- 44:26Q&A
