Running Kubernetes in Production A Million Ways to Crash Your Cluster
summary
Zalando runs 95 clusters on AWS and recounts concrete failures: a miswritten cronjob explodes pods and OOMs the API server so they enforce a 1,500 pod quota and harden the ingress controller; an alpha CSR feature piles objects into etcd and a mistaken etcdctl range delete wipes state so backups and safer CLIs are added; networking blips on T2 instances break etcd so masters move off T2; upgrade-paths break flannel so upgrades are tested by creating then updating clusters; pin Go versions and add smoke tests for registry.
AI-written summary · about
chapters (8)
- 0:00Introduction and scale
- 3:14API server outage from cron job
- 8:10etcd control delete mistake
- 10:17T2 instance network issues
- 12:02Cluster down after upgrade
- 16:39Docker registry failure
- 19:50More ways to crash clusters
- 25:06Open source and failure stories
related talks
40mContinuously Deliver your Kubernetes Infrastructure
35m10 Weird Ways to Blow Up Your Kubernetes
36m10 More Weird Ways to Blow Up Your Kubernetes
40mThe Good, the Bad, and the Ugly of Managing Production Systems with Kubernetes
41mKubernetes — A Year in Production
37mkubernetes on AWS gone wild
19mPains and gains of Kubernetes for Developers, story from experience
37m10 Ways to Shoot Yourself in the Foot with Kubernetes, #9 Will Surprise You