When it All Goes Wrong: Database Incident Response Dos and Don'ts
summary
A production Postgres cluster hits an impending transaction ID wraparound after a background vacuum stalled eight days, leaving about three hours to shutdown before the database refuses writes; responders identify the top XID consumers, shut those services, coordinate developers and support to limit impact, observe vacuum run multiple cycles over a 36 hour recovery window, fix monitoring gaps, and run targeted restores and backfill processes so the customer-facing dashboard stayed up.
AI-written summary · about
chapters (10)
- 0:00Start
- 5:39Detour: United 232 Incident
- 7:50Reasons for Survivability
- 13:21Preparedness
- 14:18General Communication Rules
- 15:22First Three Hours: Planning and Communication
- 16:58Next Three Hours: Execution
- 20:03Shortcomings
- 21:59Improvements
- 23:07Overall Assessment
related talks
33mPostgres Storytelling: Support in the Darkest Hour
51mDatabase Disasters and How to Find Them
44mBlack Monday: The Story of 5.5 Hours of Downtime
01h 07mNavigating Challenging Customer Escalations in Postgres
51mDatabase Incident Management: When Everything Goes Wrong
42mPostgres: stories that will scare you
37mWhen it all goes Wrong (with Postgres)
49mHELP!!1! The database is slow!!