Sub-Region Failure: How to Handle the Partial Loss of a Data Center
summary
Facebook claims a single failed switchboard should never force a full-region drain and proves it by choosing the switchboard fault domain, encoding per-service constraints into an auditor (Python checks plus a dashboard), and adding a power loss siren that uses rack batteries to give ~90s to fail over. A few-thousand-server outage (≈2% of a region) once cascaded because thousands of MySQL primaries triggered Zookeeper overload; after testing by flipping breakers, most switchboards no longer require region-wide drains.
AI-written summary · about
chapters (28)
- 0:00Introduction
- 0:32Hurricane Sandy
- 1:02The wakeup call
- 1:29The life of the request
- 1:47Edge points of presence
- 2:10Origin regions
- 4:12Draining regions
- 5:05Data center failures
- 6:59Subregion failures
- 7:19A few thousand servers lost power
- 8:02The switchboard
- 8:19Power panels
- 8:31Fault domain
- 9:07Drain region
- 9:51What did it take out
- 11:28The core problem
- 12:23Which services will be impacted
- 12:56Types of subregion failures
- 13:55Single fault domain
- 15:11Problem statement
- 17:21Easy services
- 18:06Constraints
- 20:18Not everything is gravy
- 21:04What happened next
- 23:28Power Loss Siren
- 24:30Power Failure
- 25:56What I learned
- 26:52Acknowledgement