Honey, I Broke the Things: Debugging Gray Failures...
summary
Slack migrates HAProxy to Envoy and hits multiple concrete regressions: a connect_failure typo that Envoy silently ignores until Envoy 1.10, missing retries (HAProxy had 3, Envoy was 0) that raises 5xxs until retries are added, host header matching that omits port causing ~25% of requests to 404, and out-of-sync TLS session ticket keys that add ~12 ms per request until synchronous ticket rotation is enforced. Tools: Envoy validate, curl, tracing at Envoy layer; fixes include stricter config generation and fast rollbacks.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 0:37Migration
- 1:33Troubleshooting
- 3:59HTTP 404
- 6:00Edge Files
- 6:58Latency
- 8:15TLS Session Ticket Key
- 9:42Lessons Learned
related talks
09mLightning Talk: Honey, I Broke the Things: Debugging Gray Failures in Production!
31mDebugging the Routing Tier
33mEnvoy on Fire: A Practical Look at Debugging a Service Mesh
28mMy Favourite Errors
21mWhen the Cloud Fails: Debugging the "Undocumented"
27mFailing forward to 1 million requests per second
09mLightning Talk: The Hardest Part of Operating a Service Mesh: Envoy Proxy
29mSafely deploying a 100K line Envoy YAML configuration to production