10 Weird Ways to Blow Up Your Kubernetes
Talkat KubeCon + CloudNativeCon North America 2019 · on Tuesday, 2019-11-19· at 15:20· Ballroom Sec 20AB - San Diego Convention Center, Upper Level·
summary
Airbnb reports running 700+ services across 1000s of nodes and dozens of clusters and demonstrates ten concrete failure modes: cron jobs that never end when sidecars live on, a kubelet patch (Lyft annotation) to order sidecars, service-mesh restarts doubling HAProxy memory, daemonsets spawning thousands of pods on a 2,000-node cluster, hitting ECR 10,000-image limit and building an ECR-cleaner singleton in a management cluster, init/sidecar ordering hacks, CRD status/observedGeneration workarounds, JVM cgroup surprises, and an HPA that spikes to ~190 pods on deploy; some fixes are hacky, some require k8s patches.
AI-written summary · about
chapters (8)
- 0:00Intro
- 0:46Zombie Jobs
- 6:12Service Mesh Speeding Accidents
- 8:52Monster Daemonsets
- 12:10Where's my docker image?!
- 18:27Where's my Custom Resource?
- 24:54Autoscale -ocalypse
- 31:14Scheduling is easy and fun
related talks
36m10 More Weird Ways to Blow Up Your Kubernetes
37m10 Ways to Shoot Yourself in the Foot with Kubernetes, #9 Will Surprise You
27mRunning Kubernetes in Production A Million Ways to Crash Your Cluster
20m10 Common Things People Do Wrong in K8s Environments
36mKubernetes Confessions: Real Mistakes We’ve All Made (and What They Teach Us)
35mKubernetes, Lessons Learnt
36m101 Ways to “Break and Recover” Kubernetes Cluster
05mReading 100+ Kubernetes Post-Mortems