QUIC Exit: Exposing a New Class of Outage
summary
A QUIC handshake null pointer in legacy code lets a Google front end issue poison encrypted tokens that clients replay to other GFEs, triggering a cascading failure that tops out at 10% of GFEs crashing and lasts about an hour and 44 minutes in this incident. Rollbacks alone fail because clients hold persistent state, four canary pushes over four minutes produce enough bad tokens, and mitigation combines emergency QUIC disable, targeted fuzzing of resumption tokens, static C++ null checks, regionalized resumption, and canary-judge changes.
AI-written summary · about
chapters (8)
- 0:00QUIC Exit outages intro
- 3:19Contagion bug discovery
- 5:44November 2021 outage timeline
- 8:57Debugging Uber proxy
- 10:21Impact and severity
- 11:45Luck in global disaster
- 12:26Long-term fixes
- 13:31Preventing contagion bugs
