Honey, I Shrunk the Data Center!
summary
Allegro runs a five‑year program called Oziris that proves losing a whole data center is survivable: teams perform controlled shutdowns, since 2023 they randomize data‑center selection with minimal notice, scope reaches over 90% of services by 2023, and they build a centralized monitoring watchtower for live health, incident and comms. Early failures include a destroyed Cassandra cluster and autoscaling exhaustion; this year they physically powered off racks with an electrician and use vendor‑integration tests after losing half user traffic when a third‑party went down.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 1:01Why simulating disasters matters
- 3:55Project Oziris and chaos engineering
- 5:43Preparing for chaos tests
- 7:12Choosing test timing
- 8:16Experiment screenplay and precision
- 10:03Monitoring the shrunken data center
- 11:08Unexpected failures and scorpions
- 12:36Project Oziris evolution
- 14:242024 lessons learned
- 15:49Project Oziris maturity
- 16:51Resilience testing side effects
- 17:55Real-world outage reveals risks
- 19:20Key learnings
- 20:21Cost and commitment
- 21:03Closing thoughts
related talks
46mChaos Experiments - Datacenter Stress Testing
42mCatastrophes and Atrophies
32mSchroedingers Backup
41mDystopia as a Service
10mBuilding an 'Oops-Less Operation' at Scale
54mAdrian’s Greatest Hits, B-Sides & Re-issues
29mTowards Sustainable Multi-Cluster Management in Real-World Academic Infrastructure
25mAutomating Disaster Recovery: The Ultimate Reliability Challenge