Scaling Kubescape to Thousands of Nodes
summary
Kubescape hits lock contention at 5,000 nodes when each node sends ~10 MB container profiles, so storing huge CRDs in ETCD becomes impractical. Initial approach of keeping full CRDs in memory and calling create every 10 minutes causes conflicts; JSON patching blocks on a synchronous 30s read/modify/write window. The re-architecture builds an aggregated API server that stores per-period CRDs as time series with a decoupled write path, acknowledges immediately, and asynchronously aggregates successfully.
AI-written summary · about
related talks
37mThe CRDs that Broke the Camel's Back
38mScaling Kube-Apiserver to Thousands of Tenants – Lessons Learned From kcp
30mScaling Kubernetes to Thousands of Nodes Across Multiple Clusters, Calmly
34mScaling Cloud Development Environments on K8s: Enterprise-Scale Adoption Lessons
32mScaling Kubernetes Application to 1M Users
15mKubernetes and etcd Features That Unlocked 15k Node Clusters
05mProject Lightning Talk: Transforming your Application Behavior to Kubernetes Objects
34mPanel: Improving and Managing Kubernetes at Scale