More Churn No Problem: Lessons Learned Running Cilium in Production
summary
Near-production integration clusters move from ENI-based flat CNI to Cilium overlay networking and eBPF, doubling usable pod density from about 90–100 to ~180 pods/node and handling churn spikes up to ~70 pod ops/sec (well over the Kubernetes SLO threshold of 20). Migration hits concrete failure modes: Cilium identity garbage-collection bug (v1.11.6) exhausts identities after weeks, intermittent node egress loss fixed by agent restarts, and API rate limits/429s during bulk pod creation that require scheduler extensions and rate-limiting knobs. Overall networking change works but needs deep Cilium tuning, monitoring and scheduler controls.
AI-written summary · about
chapters (7)
- 0:00Introduction and context
- 3:02Defining churn and SLOs
- 5:31Motivation for Cilium
- 8:47Migration strategy
- 11:33Overcoming scaling bottlenecks
- 18:19Addressing stability issues
- 24:26Conclusion and takeaways
related talks
32mSwitching your CNI — Simone Sciarrati & Federico Hernandez
24mHigh-Scale Networking for ML Workloads With Cilium
20mTackling Container Networking at Scale
16mCilium Wars: Scaling Up Your Network with Jedi-like Cilium Handling Techniques - Yash Tandon
25mThat’s Just My Cup of Tea: Configuring Cilium for Performance and Scale
26mThe Road To Cilium: Migrating 150+ Kubernetes Clusters at Airbnb
28mLive Migrating Production Clusters From Calico to Cilium
24mConfluent's Multi-Cloud Journey to Cilium: Pitfalls and Lessons Learned