10 More Weird Ways to Blow Up Your Kubernetes

on YouTube ↗36m

" summary

Test clusters hit 56,000 replica sets when a deployment’s collisionCount climbs to ~100,000 because topologySpreadConstraints aren’t copied into ReplicaSets; a mutating admission webhook produces a bad JSON patch order that prevents deploys for seven days until the webhook is removed; HPA bugs (zero current replicas, spinnaker migration, manual kubectl scale) repeatedly leave services at 0 replicas; deleting all masters breaks the API server; one deploy with huge maxSurge and crashloops drives API servers OOM; kube2iam race is mitigated by an init container that waits for creds; an ECR cleaner returns an empty list on error and deletes production images.

AI-written summary · about

" chapters (28)

1. 0:00Start
2. 1:53Context
3. 2:54Let's do more diff'ing
4. 4:20Mutating Time Bomb
5. 4:471 deployment, 3 replicasets??
6. 5:09One hour later...
7. 6:01Remember the first page?
8. 6:30The Bug
9. 9:55Edge case: ZERO
10. 13:51Runbooks to the Rescue
11. 14:3411:16, K8s master CPU goes up...
12. 15:22What is happening?
13. 15:49Response
14. 16:07Postmortem
15. 16:56Ongoing Unknowns
16. 17:50Authentication via IAM Roles
17. 19:07Race Conditions
18. 20:17More Init Containers!
19. 20:50Service Account IAM Auth
20. 21:4136 cores on machine, 36 limits on cpu
21. 21:59K8s limits pod's cpu usage using linux cgroups & CFS
22. 23:30So with 36 cores, 36 limits...
23. 24:07How is it affecting performance?
24. 25:40Oops #1
25. 26:36Oops #2 & #3
26. 27:44The Fix & Takeaways
27. 33:42Oops #3
28. 34:22The fix #2

" related talks

• 35m10 Weird Ways to Blow Up Your KubernetesMelanie Cebula, Bruce Sherrod · KubeCon + CloudNativeCon North America 2019
• 37m10 Ways to Shoot Yourself in the Foot with Kubernetes, #9 Will Surprise YouLaurent Bernaille · KubeCon + CloudNativeCon Europe 2019
• 27mRunning Kubernetes in Production A Million Ways to Crash Your ClusterHenning Jacobs · Container Camp UK 2018
• 36m101 Ways to “Break and Recover” Kubernetes ClusterSuresh Visvanathan, Nandhakumar Venkatachalam · KubeCon + CloudNativeCon Europe 2018
• 36mKubernetes Confessions: Real Mistakes We’ve All Made (and What They Teach Us)Shashank Pai · DevConf.IN 2026
• 21mHow to Not Destroy Your Production Kubernetes ClustersQian Ding · SREcon22 Asia/Pacific
• 20m10 Common Things People Do Wrong in K8s EnvironmentsKim-Norman Sahm · SREday London 2025 Q3
• 31mLoad Testing Kubernetes: How to Optimize Your Cluster Resource Allocation in Production [I]Harrison Harnisch · CloudNativeCon + KubeCon Europe 2017