Autopsy of a MySQL Automation Disaster
summary
Automated MySQL master failover using Orchestrator promotes an intermediate node X after two masters (A and B) go down, then a separate process — a DNS repointing script — later points clients at B, creating a split‑brain where writes diverge and auto‑increment IDs collide; seven simultaneous faults (hardware blade failure, premature acknowledgment, cloning the wrong server, delayed failover, missing fencing, unsafe DNS logic, and live event jobs restarting) cause data corruption. Remedies: stricter precondition checks, explicit fencing, immutable reclone workflows, and UUIDs instead of auto-increment.
AI-written summary · about
chapters (29)
- 0:00Introduction
- 1:23Team
- 1:38Outline
- 2:07Typical MySQL deployment
- 4:25Replacing an intermediate master
- 5:41Replacing a master in case of failure
- 5:51Single point of failure
- 7:05Failover
- 8:54Failure Detection
- 9:28Repointing
- 9:54Reconnecting
- 10:28Service Discovery
- 12:21War Story
- 13:04Simple Failover
- 13:30Master Service Restored
- 13:45DNS Points
- 14:24DNS
- 15:11What happened
- 15:48What caused the failure
- 16:27What made a fail
- 19:11Orchestrator acknowledgment feature
- 19:32Human error 0
- 20:38Summary
- 22:38Server failed at the same time
- 25:36How to fix such situation
- 27:24UUIDs
- 30:03Wrapup
- 31:27Questions
- 35:43Drag Drop
related talks
43mMySQL Failover for Asynchronous replication with Orchestrator, ProxySQL and Consul
05mMySQL failover with orchestrator and ProxySQL
55mProduction with ProxySQL and Orchestrator
46mDatabase Reliability Engineering for MySQL
21mHow Safe is Asynchronous Master-Master Setup?
11mThe Brave Little Database!
54mThe Evolution of a MySQL Database, From a Single Instance to HA With Disaster Recovery
52mBest Practices for MySQL High Availability in 2022