Scaling Up, One Network Bottleneck at a Time
summary
Deploys cause app RTT to spike to ~1s vs expected 100ms because startup fetches thousands of Redis keys through an Envoy sidecar that CPU-saturates, causing packet drops and TCP retransmits; Redis processes handle ~5k rps each (benchmarks show ~10x headroom), so Redis isn’t the limiter. Investigation uses proxy CPU, TCP retransmit graphs and the AWS ENA packet-drop metric over ~2 months, finds only one txq used in the kernel, applies an upstream kernel fix and then moves to a network-optimized instance to remove remaining ENA drops and restore sub-0.5s RTT.
AI-written summary · about
chapters (8)
- 0:00Identifying the scaling issue
- 2:01Analyzing high round-trip time
- 4:08Scaling the key-value store
- 5:48Proxy bottlenecks
- 10:22Kernel queue limitations
- 13:46Optimizing network bandwidth
- 16:08Key lessons learned
- 18:53Q&A and closing discussion
related talks
30mNoisy Neighbors, through Networking
25mSilicon and Networking
26mTuning Your Cloud: Improve Global Network Performance for Applications
21mMaking Envoy Resilient to Sudden Increases in Load
26mTuning Your Cloud: Improve Global Network Performance for Application
34mTroubleshooting Low Latency Application on CNF Deployment
22mAdapting TCP for Reconfigurable Datacenter Networks
44mIntroduction to application scaling