Networking Failure Stories: How to Maintain, Break, and Fix Cluster Connectivity
summary
Cilium users hit two concrete failure modes: a misconfigured network policy plus identity-label churn exhausts eBPF policy maps (cluster set to 256 entries instead of the typical 16k per-endpoint limit), causing policy-denied drops that observability tools show via Hubble flows and Cilium sysdump logs and Prometheus counters for BPF map pressure. Separately, socket-LB with kube-proxy-replacement leaves long‑lived UDP sockets pointed at stale pod IPs, causing DNS 502s; fixes: use per‑packet SNAT, disable socket‑LB, or enable forceful socket termination (requires kernel features).
AI-written summary · about
chapters (4)
- 0:00Introduction to networking failures
- 3:17First failure story connection drops
- 19:05Second failure story DNS resolution
- 28:58Lessons and best practices
related talks
25mMore Churn No Problem: Lessons Learned Running Cilium in Production
28mHow to Break your Kubernetes Cluster with Networking
24mConfluent's Multi-Cloud Journey to Cilium: Pitfalls and Lessons Learned
28mLessons from Building Scalable Network Policy Enforcement with eBPF
29m“Are These Things Talking to Each Other?” - Observing Kubernetes Networking...
10mLightning Talk: Connecting Klusters on the Edge with Deep Dive into Cilium Cluster Mesh
26mScaling Network Policy Enforcement Beyond the Cluster Boundary with Cilium
32mBreaking Boundaries: Multi-cluster with Cilium cluster mesh