Kubernetes, Lessons Learnt
summary
Kubernetes at scale shows concrete failure modes: kernel connection-tracking table exhaustion from many short‑lived TCP/UDP connections causes intermittent outages — monitor conntrack, tune via kube‑proxy params, or isolate high‑conn workloads on dedicated node groups. CoreDNS defaults (two replicas) fail at scale — use coreDNS autoscaler, pod anti‑affinity, node‑local DNS DaemonSet; one customer with ~1,000 microservices spent six months tracing DNS timeouts. Operators that repeatedly list vs watch choke the API server; prefer watch, apply API priority/fairness. Decouple control‑plane bootstrap, validate before data plane. Multi‑tenancy needs strict isolation, OPA/Kyverno and runtime hardening.
AI-written summary · about
chapters (7)
- 0:00Introduction and speaker background
- 4:10Sleeping with the Enemy: IP tables and Q proxy
- 10:32Core DNS pitfalls and scaling
- 20:47API server overload and operator impact
- 26:26Silent failures in cluster bootstrap
- 29:17Multi-tenancy risks in Kubernetes
- 33:49Final insights and Q&A
related talks
41mKubernetes the Very Hard Way
28mHow to Break your Kubernetes Cluster with Networking
36mKubernetes Confessions: Real Mistakes We’ve All Made (and What They Teach Us)
32mUnderstand and Troubleshoot the “Magic” of Kubernetes Networking - Minhan Xia & Rohit Ramkumar
31mThe Myth of the Monocluster
35m10 Weird Ways to Blow Up Your Kubernetes
06mLightning Talk: Is Your Kubernetes Cluster's DNS Working?
38mLogs Told Us It Was DNS, It Looked like DNS, It Had to Be DNS, It Wasn't DNS