The World Blew Up but We’re All Okay: How We Managed a...
summary
Datadog traces a 10+ hour global outage to a systemd change that makes systemd Networkd restart flushing routing rules, causing pod networking to drop during a coordinated security upgrade. GCP nodes lose connectivity but keep running while AWS autoscaling 60% nodes terminated in us-east-1 breaks local-disk state and causes quorum loss. Recovery sequence is parent control planes, node reboots, then massive scale-ups that hit provider limits: GCP per-instance group and subnet IP exhaustion, Vault rate-limited cert issuance and CNI retry bugs. Incident coordination scales to 70 workstreams, ~500 engineers with execs on-call and support answering a 35x ticket surge.
AI-written summary · about
chapters (6)
- 0:00The global outage
- 1:23Root cause and impact
- 7:28Incident response initiation
- 12:58Infrastructure recovery
- 23:57Managing the human factor
- 30:30Lessons and takeaways
related talks
26mThe Big One
34mDatacenter Fires and Other 'Minor' Disasters
29mKubernetes Failure Stories and How to Crash Your Clusters
26mA Tale of Two Outages
18mThe worst outage I never caused
20m[Almost] Fail & Tell: Stop The World
34m5 Things I Learnt from Causing a Cloud Provider Outage
36m101 Ways to Crash Your Cluster