Chaos Engineering:The good, the bad & the ugly
summary
XM runs chaos experiments from dev through staging/prepro to prod, using infrastructure as code and explicit stop conditions. Lifecycle: establish steady state, state a hypothesis, inject faults, collect evidence and logs, run RCA and assign an owner. Early waves took a day; mature waves run four–five experiments in ~30–45 minutes. Results: some contained incidents, occasional customer impact, and one catastrophic full-site outage when stop conditions failed. Diagrams, uniform environments and rehearsed ops teams prove decisive.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:29Who is XM
- 3:17What chaos engineering means
- 7:37Why chaos engineering
- 14:29Implementing chaos engineering
- 18:29Best practices
- 21:41Bad and ugly scenarios
- 25:57Pitfalls and fixes
- 27:27Convincing your boss
- 31:01Final warnings
- 31:42Q&A
related talks
47mChaos Engineering Bottom Up
27mChaos Engineering: The Path to Reliability
23mHow to tame chaos effectively?
18mChaos Engineering: Break It On Purpose
57mChaos Engineering Primer
37mChaos Engineering: The Fine Art of Breaking Stuff in Production on Purpose
38mChaos Engineering
36mChaos Engineering in 2021