Kubernetesception: How to run a 1000 control planes
summary
Exoscale explains how it runs Kubernetes inside Kubernetes to provision thousands of control planes fast: an orchestrator pod set generates API servers, controller managers and schedulers per customer cluster; control-plane state lives in one etcd cluster per zone to save resources; templates are deployed by a hash-based deployment so every cluster is reproducible. Surprises include a VPA tuning change that temporarily knocked out a zone and the scalability limits and noisy‑neighbor issues of a shared etcd.
AI-written summary · about
chapters (15)
- 0:00Introduction and speaker background
- 1:04Exoscale and Kubernetes at scale
- 2:27SKS architecture overview
- 3:31Customer and internal requirements
- 6:25Running Kubernetes inside Kubernetes
- 7:12Control plane provisioning details
- 11:12Tradeoffs and challenges in provisioning
- 14:43Resource management pitfalls
- 16:53State management with etcd
- 19:22etcd scalability limits and solutions
- 20:46Orchestrator lifecycle handling
- 24:04Ownership and support boundaries
- 25:34Version support and upgrades
- 27:21Key takeaways and conclusions
- 28:04Q&A session
related talks
34mScaling Kubernetes up to 1,000 Control Planes - Dario Tranchitella
35mIntro to Control Plane Engineering
37mKubeInception: running up to a thousand clusters in the real Cloud-Native way
05mkcp: Scaling The Kubernetes Control Plane For The Multi-Clust...
37mManaging and Running Multiple Kubernetes Clusters in Hybrid Setups
40mMay the Control Plane be with you: Kamaji and the rise of Kubernetes at scale
34mPanel: Improving and Managing Kubernetes at Scale
41mKubernetes the Very Hard Way