A journey to disaster
summary
Bloom and Wild treats disaster recovery as a repeatable engineering problem: they run quarterly game days that attempt to rebuild production in a new region, uncovering hardcoded regions, globally unique IAM roles and buckets, and backups landing in the wrong places. They use chaos tools (Chaos Monkey, Litmus) and automation to push secrets and multi‑region snapshots, and report progress: the first run reaches ~25% of the plan and subsequent runs cut RTO by ~0.5 business day per game day.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 3:17Disaster recovery importance
- 7:08Why prioritize DR now
- 12:44Disaster types and impacts
- 21:15Downtime costs and ripple effects
- 26:17Game day purpose
- 30:29How to run effective disaster recovery exercises
- 33:43RPO and RTO metrics
- 37:20Progress after game days
- 41:39Next steps for resilience
- 44:26Key learnings
related talks
47mChaos Engineering Bottom Up
46mEmbracing Collaborative Chaos
45mDeveloping a Chaos Architecture Mindset
14mDisaster Recovery preparedness
48mHarnessing Chaos; the hidden ingredient behind building better systems
57mChaos Engineering Primer
26mBreaking Systems for Resilience
39mResilience & Chaos Engineering in the Cloud