Limiting the Blast Radius: Preventing Outages from Bad Configs with Datacenter-Scale Health Checks
summary
Meta deploys a datacenter-scale canary called regional config validation that tests a config in one canary region for about 10 minutes and blocks global rollout if region-level errors spike. The system runs service checks then a regional gate and lands over 95% of changes under 30 minutes. It stopped a login-breaking timeout change and cuts root-cause candidates from hundreds to a handful, but many legacy or time-sensitive configs still can’t wait for a ten‑minute canary.
AI-written summary · about
chapters (11)
- 0:00Config change risks
- 1:09Canary mechanism at Meta
- 3:14Pre-canary for web front ends
- 5:01Generalizing canary checks
- 7:06Timeline of config validation
- 8:49Principles for reliable metrics
- 9:31Incident prevented by validation
- 11:15Investigating bad changes
- 14:25Expanding safety tools
- 16:30Config safety challenges
- 17:54Closing remarks
related talks
14mCanary Deployments for Infrastructure Upgrades
20mConfiguration Safety at Scale with Ads
24mCheck before You Change: Preventing Correlated Failures in Service Updates
25mPushing Code: Don't Forget to Flag Your Canaries?
42mArchitecting for Reliability
35mReduce Risks With Continuous...
20mHelp Protect Your Data Centers with Safety Constraints
35mCutting-edge Continuous Delivery: Automated Canary Analysis on CF using Spinnaker