Scaling Kube-Apiserver to Thousands of Tenants – Lessons Learned From kcp
summary
kcp runs virtual‑user load tests simulating 10,000 workspaces with a custom framework (couldn’t use PerfTest) and sharded etcd to avoid single‑node bottlenecks. Tests expose two root problems: per‑workspace garbage‑collector state and the resource‑quota controller creating informers per GVR, causing spikes (≈70 GB/shard) and high CPU. Fixes include a single DAG keyed by (UID,cluster) and a dynamic shared informer factory, yielding about 66% memory reduction and far lower informer churn.
AI-written summary · about
chapters (8)
- 0:00Introduction and motivation
- 1:52What is KCP
- 5:23Test setup and methodology
- 15:18Load testing results
- 21:43Fixing garbage collector overhead
- 23:09Resource quota controller issues
- 25:14Multi-tenancy performance insights
- 27:43Future work and community call
related talks
41mKcp: Towards 1,000,000 Clusters, Name^WWorkspaced CRDs
05mkcp: Scaling The Kubernetes Control Plane For The Multi-Clust...
37mOne Cluster to Rule Them All - Multi-Tenancy for 1k Users
22mTaming the VM Sprawl: KCP Meets KubeVirt
37m73,000 Pods a Day, Lessons From Misadventures In Multi-Tenant
34mScaling Cloud Development Environments on K8s: Enterprise-Scale Adoption Lessons
32mScaling Kubernetes Application to 1M Users
01h 15mTutorial: Exploring Multi-Tenant Kubernetes APIs and Controllers With Kcp