Scaling in Kubernetes Safely on On-Prem KaaS Across 1,300+ Clusters and 40,000+ Nodes
summary
A custom controller consolidates underutilized worker nodes across 1,300+ clusters and 40,000+ nodes, reducing capacity by about 20% so far with zero production incidents. Decisions use observed usage, per-pod requests, and cluster-wide metrics: 7-day peak <30%, request/allocatable <50%, and keep ≥6 nodes across 3 AZs, with a 3-day cooldown after each reduction. Guardrails include ordered DaemonSet termination, graceful-shutdown delays, PDB and resource-request validation, ephemeral-storage sizing (include unpacked image size), Grafana dashboards, resource annotations and Kubernetes events for auditability.
AI-written summary · about
chapters (19)
- 0:00Introduction
- 1:25Platform background
- 3:39Automated node failure recovery
- 5:44Production-ready clusters
- 7:09Hardware cost pressures
- 9:19Unused capacity challenges
- 10:50Low usage node reducer
- 12:43Safety checks for node removal
- 15:00Graceful shutdown sequence
- 16:51Shutdown order for components
- 17:54Distribution budget pitfalls
- 18:40Service resource requests
- 19:49Ephemeral storage monitoring
- 20:33Large container images
- 21:15Guardrails for safety
- 22:43Rolling node deployment
- 23:44Scaling results and incidents
- 24:26Key takeaways
- 25:28Q&A
related talks
36mOptimizing Resource Usage in Kubernetes
15mDemo: Resource(Cost) optimization on planet scale Kubernetes clusters.
32mScaling Kubernetes Application to 1M Users
29mRight-Size Your Kubernetes Deployments
17mScaling Made Simple: Implementing Autoscaling for Optimal K8s Performance
34mPanel: Improving and Managing Kubernetes at Scale
11mCost and Resource Optimization In Kubernetes
39mOptimizing Resource Usage in Kubernetes