Disaster Recovery Testing at Booking.com
summary
Booking.com documents a program that starts with 80,000 servers and ~2,000 services and moves from a manual region evacuation that takes 4+ hours to seconds after automation; initial drill needs ~300 people, later reduces to one-button failovers and weekly automated evacuations. Tests include injected latency, packet drops, cutting inter-datacenter links and power, and reveal concrete failures: VPN outages, cross-region health-check blind spots, MySQL replication split-brain and long restore windows for stateful services, prompting playbooks, execution plans and staged restarts.
AI-written summary · about
chapters (14)
- 0:00Intro
- 2:25Story
- 5:22Risk
- 5:45First Failover Test
- 7:16Challenges
- 8:21Growing the program
- 9:51Execution plans
- 11:47Benefits
- 15:09People
- 16:37Disaster Recovery
- 21:29Fully Isolated
- 24:49Power Down
- 27:47Public Cloud
- 29:13Key takeaways
related talks
22mEvolution of Disaster Recovery @ Meta
29mBuilding Resilient Systems
53mResiliency and Availability Design Patterns for the Cloud
33mThe DiRT on Chaos Engineering at Google
10mBuilding an 'Oops-Less Operation' at Scale
42mChaos Engineering: Why Breaking Things Should be Practiced
44mAvailability, Latency and Cost: Withstanding Outages
57mChaos Engineering Primer