101 Ways to Crash Your Cluster
summary
Series of Nordstrom postmortems that name concrete failure modes and fixes: Prometheus plus bursty Flink jobs cause page allocation stalls and OOM kills resolved by kubelet eviction thresholds and node systemReserved/kubeReserved tuning; AWS ELB TCP timeouts flip 25–50% of nodes NotReady for ~15 minutes fixed by switching to NLB or upgrading to 1.7.8; cluster-autoscaler once shrinks a cluster to ~1/3 capacity so smoothing and an observability-upgrade are added; etcd returns stale reads causing a four‑hour outage mitigated by quorum reads (1.9+).
AI-written summary · about
chapters (11)
- 0:00Start
- 2:54The Tale of the Unresponsive Node
- 3:46Looking into the past
- 6:11Not Ready Node Troubleshooting Steps
- 6:36Set your kubelet flags correctly
- 8:30NotReady Storm Checklist
- 10:20Following the trail of clues
- 11:59Fixes for the Node NotReady Storm
- 14:24All the nodes went away checklist
- 15:17The Cluster Autoscaler Robots
- 21:39Split Personality etcd Cluster
related talks
29mKubernetes Failure Stories and How to Crash Your Clusters
39m15 KUBERNETES FAILURE POINTS YOU SHOULD WATCH
35mKubernetes, Lessons Learnt
34mReading 100+ Kubernetes Post-Mortems - lessons learned
20mYou can't have a cluster*!$& without a cluster!
30mWhat we learned from reading 100+ Kubernetes Post-Mortems
51mHearts of Darkness: A Spring DevOps Apocalypse
37mRabbitMQ has a problem - Let us tell you about it