A Tale of Two Outages
summary
A 2009 outage destroys a single SAN’s metadata partition, leaving a 15 TB relational database unreadable until a third‑party recovery team reconstructs 4K pages by GOID over six weeks using a hastily assembled Windows “Lazarus” fleet and recovers three nines of data; an unnamed 2018 global sale overloads a NoSQL backend where pages require hundreds to thousands of DB calls, a routing/caching layer of >1,000 servers and a 1 MB TCP receive window hide queued transactions that fuel retry storms and force hours of hotfixes and throttling to restore service.
AI-written summary · about
chapters (10)
- 0:00Introduction and Speaker Background
- 1:29The Dangerous Sidekick Outage: Setting the Scene
- 3:16The Dangerous Sidekick Outage: The Disaster Unfolds
- 5:59The Dangerous Sidekick Outage: Recovery Efforts
- 10:02Lessons Learned from the Sidekick Outage
- 11:30The E-commerce Giant Outage: Setting the Scene
- 13:18The E-commerce Giant Outage: The Disaster Unfolds
- 17:34The E-commerce Giant Outage: Recovery Efforts
- 21:45Lessons Learned from the E-commerce Outage
- 23:45Final Takeaways and Q&A
related talks
27mLife in the Trenches of healthcare.gov
34mDatacenter Fires and Other 'Minor' Disasters
40mThe World Blew Up but We’re All Okay: How We Managed a...
26mThe Sounds of Silence: Lessons from an 18 hour API outage
25m"The website's down!" Stories and lessons on keeping your website up.
34m5 Things I Learnt from Causing a Cloud Provider Outage
44mBlack Monday: The Story of 5.5 Hours of Downtime
43mDisaster Non-Recovery