Lightning Talk: Honey, I Broke the Things: Debugging Gray Failures in Production!
summary
Migration from HAProxy to Envoy hits three concrete failures: a config typo ('connect_ failure' vs 'connect-failure') that Envoy ignored so retries never applied — fixed by validating allowed strings and updating tests; a canary at 25% traffic that exposed requests with host headers like "host:443" causing 404s until edge routing matched host:port and a rollback eased customers; and a TLS session-ticket key rotation bug that adds ~12 ms per request (observed as ~4% latency) — fixed by synchronous key rotation after tracing at the Envoy layer.
AI-written summary · about
related talks
13mHoney, I Broke the Things: Debugging Gray Failures...
09mLightning Talk: The Hardest Part of Operating a Service Mesh: Envoy Proxy
33mEnvoy on Fire: A Practical Look at Debugging a Service Mesh
21mWhen the Cloud Fails: Debugging the "Undocumented"
31mDebugging the Routing Tier
27mFailing forward to 1 million requests per second
01h 25mCloud-Native Building Blocks: An Interactive Envoy Proxy Workshop
29mSafely deploying a 100K line Envoy YAML configuration to production