Deconstructing an Abstraction to Reconstruct an Outage
summary
At 15:09 a RAID controller ejects disks and the primary goes down; engineers reproduce a three-node Postgres cluster in Docker Compose, inject faults with SIGKILL and crash the synchronous replica, and even hex-edit WAL records to set the xlog total_len to zero, which triggers an "invalid record" error but does not cause failover. The real trigger is Pacemaker config: a backup VIP colocated on the replica plus a negative infinity collocation prevents rescheduling, leaving the cluster stuck.
AI-written summary · about
chapters (21)
- 0:00Introduction to the outage story
- 1:48Speaker introduction and background
- 2:50Cluster architecture and setup explanation
- 4:14The incident begins with RAID controller failure
- 6:00Mission to recreate the outage
- 7:48Initial fault injection attempts
- 9:58Understanding write-ahead logs in PostgreSQL
- 13:49Debugging the binary log error message
- 16:15Analyzing binary log files in a hex editor
- 19:28Modifying binary logs to reproduce the error
- 20:57Discovering the backup virtual IP address role
- 22:41Pacemaker configuration and constraints
- 25:27The write-ahead log error as a red herring
- 26:31Minimal reproduction of the outage
- 26:51Lessons learned from the incident
- 27:54The danger of automation eroding knowledge
- 28:58Importance of game days and manual practice
- 29:41Questioning reproduction and wider context
- 30:02Prediction about binary formats in web development
- 31:27Learning from outages and incident reviews
- 32:55Call to share difficult incident stories
related talks
18mChaos testing of a Postgres cluster on Kubernetes
38mWhen it All Goes Wrong: Database Incident Response Dos and Don'ts
42mPostgres: stories that will scare you
33mFun with Postgres High Availability Poker
55mHigh Availability with PostgreSQL and Pacemaker
54mBreakout Session
51mDatabase Disasters and How to Find Them
45mRecovering from a damaged cluster