What I Learnt Fixing 50+ Broken Kubernetes Clusters
summary
A livestream series called Klustered fixes over 50 intentionally broken Kubernetes clusters and distills nine practical operator lessons: surprises include kubelet downgrades by 20–30 versions, a compiled API server injected into control planes, and immutable log directories. Fixes use OS tricks (running ld‑linux.so to bypass missing +x), forensics with fls/SwiftKit to recover file changes, and Cilium/Hubble to trace diverted Postgres traffic. eBPF and tools like Falco, Pixie and Inspector Gadget provide powerful visibility but can also be abused (Hubble, eBPF).
AI-written summary · about
chapters (15)
- 0:00Introduction
- 1:09Clustered live stream origin
- 4:24Debugging a Clustered clip
- 9:29Kubernetes failure lessons
- 10:16Linux fundamentals for K8s
- 13:45Fixing permissions without chmod
- 14:52Intrusion detection with Swift/fls
- 17:18Kubernetes networking pitfalls
- 24:23Safe etcd operations
- 28:37Container runtime security risks
- 31:44Dismantling hero culture
- 32:26eBPF for Kubernetes observability
- 37:21Demo and challenges
- 41:55Kubernetes risk pyramid with eBPF
- 42:17Q&A: Tetragon and service mesh
related talks
33mKubernetes Security: Horror Stories to Help Secure Your Cluster
36m101 Ways to “Break and Recover” Kubernetes Cluster
36mKubernetes Confessions: Real Mistakes We’ve All Made (and What They Teach Us)
35m10 Weird Ways to Blow Up Your Kubernetes
32mSurviving From Endless Issues Coming From 7K+ Kubernetes Clusters
42mMastering Kubernetes: Challenges, Strategies, and Best Practices
27mA Safari of Kubernetes and its Natural Habitat
34mReading 100+ Kubernetes Post-Mortems - lessons learned