Embracing Collaborative Chaos: Chaos Days on Large Platforms
summary
Large government platform running hundreds of microservices across six platform teams runs deliberate Chaos Days to expose real failure modes, using agents of chaos coordinating in private Slack and Trello, and even a hack that killed Kubernetes pods triggered from a game emulator. Experiments run in production-like pre-prod or live AWS, typically five to ten experiments per event, include simulated AZ loss and alerting failure, and produce concrete outcomes — rollbacks, improved runbooks, post-incident reviews and measurable resilience after handling spikes like 140,000 claims on launch day.
AI-written summary · about
chapters (25)
- 0:00Intro
- 1:08Building vital, high traffic services, fast
- 2:44Building planet-scale, services, fast
- 3:44Operating on the edge of chaos
- 5:13How can your system fail?
- 6:40Addressing the risk of failure
- 8:09resilience
- 9:07Four chaos engineering approaches
- 9:30Manual chaos
- 10:43Automated chaos
- 13:13Unplanned chaos
- 15:29How does it help?
- 18:34Our context
- 19:51When were we ready for chaos?
- 22:37Agents of chaos
- 23:46Chaos scope - know thyself
- 25:17Chaos scope - trust the brains-storm
- 25:43Chaos scope - brainstorm, then plan the detail
- 28:26Chaos scope - hack the chaos
- 29:14Deciding where
- 32:59Keep calm and chaos on (agents)
- 34:17Keep calm and chaos on (everyone else)
- 35:52Divide and conquer, then regroup
- 37:35What did we learn?
- 39:52What's your next chaos step?
related talks
46mEmbracing Collaborative Chaos
01h 01mEmbrace chaos to achieve stability
47mChaos Engineering Bottom Up
19mChaos Engineering at Scale
01h 04mTrust and Confidence through Chaos
40mJourney into the World of Chaos Engineering
48mHarnessing Chaos; the hidden ingredient behind building better systems
39mResilience & Chaos Engineering in the Cloud