Thirty seconds of silence
summary
A production Kubernetes cluster experiences 30 seconds where new pods cannot pull images across roughly 50–80 pods affected; containerd shows connection refused to an internal IP (10.1.0.89). Root cause is stale endpoints caused by kube-proxy’s iptables flush every 10s combined with eviction timing so nodes try an evicted ingress-nginx pod IP. Fix is using ingress-nginx’s graceful shutdown via a preStop hook + 10s sleep, plus extra telemetry; the mitigation holds for two years with no recurrence.
AI-written summary · about
chapters (15)
- 0:00Introduction and context
- 1:31What is Octeto platform
- 2:55The 30 seconds of Silence incident
- 4:19Initial investigation begins
- 11:25Brainstorming and team effort
- 13:16Timeline of the event
- 13:37ContainerD logs reveal clues
- 18:17DNS and IP resolution check
- 19:00Kubernetes service discovery basics
- 21:51Kube-proxy IP table updates
- 22:32Root cause analysis
- 24:24Solution and prevention
- 25:27Pre-stop hooks in Kubernetes
- 27:58Testing and validation
- 29:24Key takeaways and conclusion
related talks
38mCaulking the deployment gap: absolutely zero downtime deployments in Kubernetes
33mThe Gotchas of Zero-Downtime Traffic /w Kubernetes
28mHow to Break your Kubernetes Cluster with Networking
37m10 Ways to Shoot Yourself in the Foot with Kubernetes, #9 Will Surprise You
30mThe Unconspicuous Role of Conntrack in Kubernetes Networking
34mWhere's My Pod, Already? A Tale of Cold-Start Struggles on Kubernetes
38mLogs Told Us It Was DNS, It Looked like DNS, It Had to Be DNS, It Wasn't DNS
35mKubernetes, Lessons Learnt