Taming the BPF LRU: Eliminating TCP Resets in Cilium

on YouTube ↗10m

" summary

At scale Cilium’s BPF SNAT plus an approximate LRU eviction policy is dropping active NAT entries and injecting wrong source ports, causing TCP RSTs; diagnosis uses tcpdump and PWU to trace packets to the tailcall helper and the reverse-NAT code path. The merged fix in Pull Request #37747 proactively restores original NAT entry from the reverse entry, reducing failures from single-digit percentages (reported up to 10%) to nearly zero in tests.

AI-written summary · about

" chapters (9)

1. 0:00BPF LRU and TCP resets
2. 1:06TCP resets in production
3. 2:34Unexpected port changes with tcpdump
4. 3:37BPF tracepoints for issue pinpoint
5. 5:48BPF LRU eviction logic
6. 7:14Reverse entry recovery fix
7. 8:15Fix effectiveness with tests
8. 9:20Dynamic map sizing
9. 10:01Acknowledgments

" related talks

• 22mTales from an eBPF Program’s Murder MysteryHemanth Malla, Guillaume Fournier · CiliumCon Europe 2023
• 19mTackling DC Congestion and BurstsBalasubramanian Madhavan, Abhishek Dhamija · Networking @Scale Summer 2022
• 38mUnderstanding and Troubleshooting the eBPF Datapath in CiliumNathan Sweet · KubeCon + CloudNativeCon North America 2019
• 31mNetworking Failure Stories: How to Maintain, Break, and Fix Cluster ConnectivityTomasz Tarczyński · ContainerDays Hamburg 2026
• 14mA large-scale deployment of DCTCPBalasubramanian Madhavan, Abhishek Dhamija, Prashanth Kannan, Neil Spring, Shrikrishna Khare, Hechao Li, Jie Meng, Madhavi Rao, Lawrence Brakmo, Srikanth Sundaresan, Soudeh Ghorbani · 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI '24)
• 44mCilium - Container Security and Networking Using BPF and XDPThomas Graf · Open Source Summit North America 2017
• 31mUnleashing the Tides of Kubernetes Networking by Removing kube-proxyTomasz Tarczyński, Filip Wardzichowski · ContainerDays Hamburg 2026
• 07mLightning Talk: Taming Noisy Network Neigbours with Cilium and eBPFAnton Ippolitov · Cilium + eBPF Day Europe 2024