The DiRT on Chaos Engineering at Google
summary
Google reports 15 years of Disaster Resiliency Testing with thousands of planned experiments and teams now running tens of thousands of tests per year to prove hypotheses. The program uses an rpc fault injection library to inject latency and errors, a practice that includes burn the SLO quarterly on Chubby to force consumer resilience. Tests ramp blast radius, monitor error-budget burn, and rely on a big red button rollback and automated repeatable tests; results shorten real incident mitigation times.
AI-written summary · about
chapters (17)
- 0:00Intro
- 1:02DiRT: Disaster Resiliency Testing
- 2:53Why?
- 4:38What we test?
- 6:01Testing themes
- 10:01Practical vs theoretical
- 12:31How?
- 15:12Picking what to test
- 16:29Steps for bootstrapping a disaster testing program
- 18:25Testing production vs testin in production
- 20:16Really, you're breaking production though?!
- 23:00Reporting on results
- 24:24What have we learned?
- 26:55Test example: Run at service level
- 28:51Test example: Toggle the O-N / O-F-F discriminator
- 30:25Test example: Run without dependencies
- 31:53Test example: Hacked!
related talks
29mThe Road to Reliability
48mChaos Engineering: The Path to Reliability
26mFrom experimentation to continuous verification
26mBreaking Systems for Resilience
27mChaos Engineering: The Path to Reliability
18mChaos Engineering: Break It On Purpose
57mChaos Engineering Primer
39mResilience & Chaos Engineering in the Cloud