The worst outage I never caused
summary
A routine config rollout almost drops Google’s global prefixes when an unintended change removes a crucial magic number, but the deploy tool’s delay and a quick revert prevent what would have taken full effect in two to three seconds. Post-mortem work reveals hundreds of commented-out lines, no ownership, and no tests, so a set of validation checks and a Python generator (protobuf input → canonical config) is built to enforce route counts, unique prefixes and correct flags, yet management buy-in for broader cleanup remains limited.
AI-written summary · about
