Preventing Controller Sprawl From Taking Down Your Cluster - When a Scalable Pattern Stops Being Scalable
summary
Controllers rely on expensive list+watch semantics so heavy mutation rates break guarantees: a buggy change to the in-house archetype adds conflicting service-account annotation churn, Calico (with Typha) falls behind and deletes routes causing 2–10 minute blackouts for new pods; in a separate incident a burst creating thousands of pods in minutes makes the API server’s watch cache evict history, returning HTTP 410s so thousands of kubelets restart watches, which spirals the control plane CPU and crashes API servers. Monitoring, audit logs, and code-reading fix and mitigate these failures.
AI-written summary · about
chapters (23)
- 0:00Intro
- 2:52Kubernetes philosophy
- 5:38Kubernetes controllers
- 9:35Example: Pizza controller
- 11:59Case Study: #1
- 12:44It's the network, stupid
- 14:16Calico for L4 segmentation
- 17:08Routing problem?
- 18:36High ping latencies
- 20:41A-HA moment!
- 23:12Battle of the controllers
- 25:02Why did this affect Calico?
- 27:31Case study: #2
- 29:27Off the charts
- 29:59Theories and temporary mitigation
- 32:00Do you even HTTP 410
- 34:22Downward spiral
- 36:07Takeaway: Observability - Metrics
- 38:07Takeaway: Observability - Logs
- 39:31Takeaway: Observability - Audit logs
- 40:24Takeaway: Change visibility
- 41:47Takeaway: Read the code!
- 42:45Takeaways: Read the code!
related talks
20mYou can't have a cluster*!$& without a cluster!
32mDon't Write Controllers Like Charlie Don't Does: Avoiding Common Kubernetes Controller Mistakes
28mScaling in Kubernetes Safely on On-Prem KaaS Across 1,300+ Clusters and 40,000+ Nodes
35mHandling Risky Business: Cluster Upgrades
35mKubernetes, Lessons Learnt
35mIntro to Control Plane Engineering
31mBeyond the Limits: Scaling Kubernetes Controllers Horizontally
46mOperators Live in the Past: Designing Reliable Kubernetes Controllers