The CRDs that Broke the Camel's Back
summary
Installing ~2,000 CRDs causes two failures: client-side discovery hits a token‑bucket limiter (default burst 300, rate 50 qps; older bug used 100/5) so cold-cache discovery on ~370 groupVersions triggers throttling, and the API server spends CPU on openapi/v2 aggregation JSON (heavy allocs and GC). Reproduced with custom kube-apiserver builds and pprof on kind; upstream fix defers aggregated spec computation until request and cuts CPU peaks, but managed EKS/AKS/GKE behavior still varies.
AI-written summary · about
chapters (5)
- 0:00Start
- 5:16Client Side Throttling Issues
- 5:44Warning Messages
- 28:24Cpu Utilization
- 31:35Profile the Api Server
related talks
05mScaling Kubescape to Thousands of Nodes
36m10 More Weird Ways to Blow Up Your Kubernetes
29mKubernetes Reloaded: AKA How to Avoid the Dev Plateau with CRDs
38mScaling Kube-Apiserver to Thousands of Tenants – Lessons Learned From kcp
26mTroubleshooting Kubernetes CRDs is too damn hard
35m10 Weird Ways to Blow Up Your Kubernetes
29mKubernetes Reloaded AKA How to Avoid the Dev Plateau with CRDs
27mRunning Kubernetes in Production A Million Ways to Crash Your Cluster