73,000 Pods a Day, Lessons From Misadventures In Multi-Tenant
summary
Claim: thinking in CPU time not cores fixes massive multi‑tenant surprises. After instrumenting clusters with Prometheus and cAdvisor and querying container_cpu_usage_seconds_total they discover CPU throttling and node saturation pressure (memory and IO stalls) caused churn of ~73,000 pods/day. They replace percent‑of‑requests autoscaling with a custom HPA driven by active thread counts (PromQL dashboards and cgroup/CFS insights), cut needless scaling, and stabilize production. Kernel CFS behavior, cgroups v2 limits, and correct PromQL queries are the actionable tools.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 1:31Kubernetes and Linux mismatch
- 2:29Understanding CPU limits
- 4:47Measuring time-based usage
- 7:01Real-world performance impact
- 8:35Limits vs requests
- 10:40The Linux scheduler
- 13:06CFS and IO interaction
- 15:49Node saturation
- 19:59Memory and IO pressures
- 22:05Rethinking HPA metrics
- 26:10The fallacy of scaling up
- 29:16Scaling alternatives
- 32:07Finding the right metric
- 33:47Conclusion
related talks
44mFinOps at scale: Lessons from scaling Kubernetes workloads
21mLet your Kubernetes environment flourish
31mFailing to Autoscale?
38mScaling Kube-Apiserver to Thousands of Tenants – Lessons Learned From kcp
47mOptimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and Latency
36mKubernetes Confessions: Real Mistakes We’ve All Made (and What They Teach Us)
15mOptimizing Application Performance on Kubernetes
38mA Tale of Tail Latency: Understanding k18n CPU Requests and Limits for Sust. and Profit