The worst outage I never caused
summary
A 2017 change comes one keypress from dropping Google's backbone after a magic-number mismatch in BGP policies. Root causes are obsolete config, no content tests, and fuzzy ownership. A quick solution builds a simple python testing toolkit that parses configs from text protocol buffer, generates canonical files, and runs diff and uniqueness checks. The lightweight generator and tests catch latent errors, validate route semantics, and prevent the outage in practice.
AI-written summary · about
chapters (6)
- 0:00Introduction
- 1:28The incident
- 2:39What happened
- 5:46How did we get there
- 9:37Root causes
- 12:10Postmortem
related talks
18mThe worst outage I never caused- 18mThe worst outage I never caused
39mOops. I broke the Google, now what?
22mWhy is it always DNS, TLS, and bad configs?
40mThe World Blew Up but We’re All Okay: How We Managed a...
35mRoute to Bugs: Analyzing the Security of BGP Message Parsing
22mBGP Vortex: Update Message Floods Can Create Internet Instabilities
41mSleeping with one eye open: production support