Black Monday: The Story of 5.5 Hours of Downtime
summary
A postmortem of a major production outage that caused 5.5 hours of downtime, analyzing causes, team response, and mitigation steps to prevent recurrence. Focuses on root-cause investigation with special attention to PostgreSQL’s VACUUM mechanism and its role in the incident.
AI-written summary · about
chapters (9)
- 0:00Worst day in history
- 7:46Customer analytics solution
- 13:55Downtime begins
- 18:12System recovery
- 22:14Client communication
- 25:46Cause: excessive access rights
- 29:02Lessons: backup plans
- 36:09Tech choices
- 42:16Post-incident testing
related talks
38mWhen it All Goes Wrong: Database Incident Response Dos and Don'ts
26mThe Sounds of Silence: Lessons from an 18 hour API outage
34mDatacenter Fires and Other 'Minor' Disasters
21mHandling a Major Outage Transparently
42mPostgres: stories that will scare you
51mDatabase Disasters and How to Find Them
33mPostgres Storytelling: Support in the Darkest Hour
26mA Tale of Two Outages