The day I broke production and it was on national television
summary
A midnight maintenance to split a four-node cluster into two fails, triggering a split‑brain so neither half forwards multicast traffic; rollback also fails because a multicast bug hides failures from unicast-based monitoring. Control room and a provider with a backup path keep half the country watching while about a 45-minute outage hits others; a full reboot (about a 10-minute reboot per side) restores service. Lesson: lab had a two-node model, production behaved differently, so split the cluster and insist on accurate multicast-aware monitoring.
AI-written summary · about
chapters (20)
- 0:00Production disaster story
- 3:59Company and network setup
- 8:12Multicast service ownership
- 10:22Cluster split plan
- 12:30Clearing technical debt
- 13:13Maintenance at 2 AM
- 14:36Failure during config changes
- 15:19Lab vs production gap
- 16:24Flawed rollback plan
- 17:08Unreliable monitoring
- 18:11Escalating to lead engineer
- 18:56Outage on national TV
- 19:38Client backup communication
- 20:19Rebooting cluster
- 21:04The importance of having someone to talk to
- 21:48Revisiting the change
- 22:32Supportive manager value
- 23:13Root causes beyond error
- 23:53Multicast networking complexities
- 24:15Incident analysis lessons
related talks
50mOps issues
33mDatacenter Fires and Other "Minor" Disasters
20m[Almost] Fail & Tell: Stop The World
34mDatacenter Fires and Other 'Minor' Disasters
37mEffective Disaster Recovery: The Day We Deleted Production
07mLightning Talk: Mission: Impossible - Building Self-Healing Disaster Recovery With...
40mThe World Blew Up but We’re All Okay: How We Managed a...
45mWhat I Learnt Fixing 50+ Broken Kubernetes Clusters