Lessons Learned from the Halloween Outage
summary
73-hour global outage at Roblox stems from enabling Consul streaming on load balancers which, 24 hours later, triggers a Consul performance collapse and then an open-source storage library bug under extreme backlog; core facts: single operational core data center, 20M RPS at the front door, 120M RPS cache, 23 edge DCs. Timeline: 50 hours to identify and roll back the Consul change, 4 hours for a workaround, 19 hours to relaunch while building a DNS steering ramp; fixes: roll back first, rotations, cells and turn-up automation.
AI-written summary · about
chapters (6)
- 0:00Intro
- 1:47Roblox Architecture
- 7:56Hero Mode Is Dangerous
- 11:41The Importance Of Infinite Undo
- 15:12Graceful Degradation Has Its Limits
- 20:31Push Button, Get Roblox
related talks
26mThe Sounds of Silence: Lessons from an 18 hour API outage
28mReliability Lessons from Running High Traffic Services for More Than 10 Years
40mThe World Blew Up but We’re All Okay: How We Managed a...
26mA Tale of Two Outages
34mDatacenter Fires and Other 'Minor' Disasters
18mThe worst outage I never caused
47mA Transformation Story: How We Failed at 20,000+ RPS and What We Learned
25m"The website's down!" Stories and lessons on keeping your website up.