[Almost] Fail & Tell: Stop The World
summary
Meta avoids a major site outage by containing impact to one region and restoring service within about 15–20 minutes. Recovery re-bootstraps the regional control plane by finding hosts with old binaries, running recovery containers via the Bell Jar toolkit, and reverting a staged feature flag that caused schedulers to stop. Only a handful of zeroth-shard jobs needed manual restart, after which normal schedulers resumed. Postmortem actions include promoting Zelos to a scheduler peer, richer historical logs, and more frequent Bel Jar drills.
AI-written summary · about
chapters (9)
- 0:00Meta incident intro
- 1:51Incident overview and response
- 4:38Twine and root cause
- 8:06The bug and impact
- 9:50Initial containment steps
- 11:54Bell Jar toolkit recovery
- 14:22Restoring site operation
- 15:48Follow-up and architecture changes
- 17:54Lessons learned
related talks
40mThe World Blew Up but We’re All Okay: How We Managed a...
07mLightning Talk: Mission: Impossible - Building Self-Healing Disaster Recovery With...
48mHow a Couple of Characters (and GitOps) Brought Down Our Site
41mHave You Tried Turning It off and Turning It on Again?
26mThe Big One
39m30 minutes of Failure
25mThe day I broke production and it was on national television
25mAutomating Disaster Recovery: The Ultimate Reliability Challenge