No Requests, No Limits, No Service How to Bring Kubernetes to a Grinding Halt
summary
Octopus Deploy’s internal test Runner floods a Kubernetes cluster causing 750 waiting pods and kubelet failing to contact containerd; first incident on 25 Aug 2022, second on 29 Nov 2023. Key contributing factors: small nodes with spinning disks, imagePullPolicy: Always triggering massive registry pulls, a retry pattern that cascades scheduling across nodes, and many pods launched without resource requests. Remediations include reducing maxPods 250→125, adding requests, SSDs, and slower retries, but systemic drift keeps similar failures possible.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 0:43Octopus Deploy methods
- 1:26SSH to Kubernetes pods
- 2:29First outage Aug 25 2022
- 4:58Root causes of outage
- 6:22Second outage Nov 29 2023
- 7:51Why second incident happened
- 8:52Complex systems and failures
- 10:18Beyond root causes
- 12:06Drift into failure explained
- 13:21Reasonable decisions problematic
- 14:26Preventing failures in systems
- 15:10Key takeaways and rule of thirds
- 15:54Investing in success factors
- 16:15Closing remarks
related talks
36mKubernetes Confessions: Real Mistakes We’ve All Made (and What They Teach Us)
05mReading 100+ Kubernetes Post-Mortems
33mWhat We Learned from Reading 100+ Kubernetes Post-Mortems
31mThe Myth of the Monocluster
33mThe Gotchas of Zero-Downtime Traffic /w Kubernetes
02h 26mKubernetes Performance Tuning Workshop
36mOptimizing Resource Usage in Kubernetes
28mScaling in Kubernetes Safely on On-Prem KaaS Across 1,300+ Clusters and 40,000+ Nodes