Envoy’s Using 10GB of Memory and It’s All My Fault!
summary
Custom xDS server misorders and floods Envoy with TLS listener updates, causing linear memory growth up to ~70 GB (30 GB in another run) when applying 5,000 ingresses in blocks; old listeners accumulate in draining mode with a 600s drain timeout and hold memory. Reproduction uses zero traffic, Kubernetes watches + client-go, and Contour as the xDS server. Mitigations that reduce but don’t fully fix it are sorting secrets before xDS push, implementing XDS ACKs/discovery request handling, and batching rapid cluster events.
AI-written summary · about
chapters (8)
- 0:00Intro
- 0:30The Issue
- 1:44What is Contour
- 2:42Contour Implementation
- 3:44Memory Growth
- 4:45Recap
- 5:47The Problem
- 7:12Current Solution
related talks
28mEnvoy Project Intro
34mIntro: Envoy
08mLightning Talk: It’s 10pm, do you know what your proxy is doing? Understanding...
38mEnvoy Deep Dive
27mEnvoy - Overview and Update: Senior Envoy Maintainer
37mEnvoy Internals Deep Dive
25mAdapting Our XDS Control Plane for Proxyless gRPC
21mMaking Envoy Resilient to Sudden Increases in Load