Scale Kubernetes to Support 50,000 Services [I]
summary
Demonstrates replacing iptables in kube-proxy with IPVS mode and adding a half-second batching of endpoint updates to cut API-server QPS, turning hours-long rule updates into millisecond updates. Prototypes add ~680 lines to kube-proxy, show a 98% QPS drop from batching, reveal iptables can take 11 minutes per rule at scale, and validate IPVS handles tens of thousands of services (benchmarks reach 50,000 services with per-update latency near constant and throughput degrading only ~2x).
AI-written summary · about
chapters (15)
- 0:00Intro
- 1:51What are the Challenges while Scaling Services
- 6:35Control Plane Solution Alternatives
- 9:06Batch Processing Request Reduction
- 9:58Batch Processing End-to-End Latency Reduction
- 12:33IPTables Example
- 13:06IPTables Service Routing Performance
- 14:22Latency to Add IPTables Rules
- 16:08Data Plane Solution
- 17:23Restructure IPTables as a Search Tree
- 18:03What is IP Virtual Server (IPVS)?
- 20:12IPVS Load Balancing Mode in Kubernetes
- 21:29IPVS vs. IPTables Latency to Add Rules
- 22:41IPVS vs. IPTables Network Bandwidth
- 24:07Other PertScalability Work We Have Done.
related talks
34mLatest Kubernetes Scalability Improvements
35mLiberating Kubernetes From Kube-proxy and Iptables
36mUnderstanding How kube-proxy Works in IPVS Mode
29mWhy We Ditched Kube-proxy: Scaling 10M Daily Browser Sessions With Kubernetes Endpoi...
51mKubernetes SIG-Network: Intro and Deep-Dive
31mUnleashing the Tides of Kubernetes Networking by Removing kube-proxy
27mInside Kubernetes Services
34mPanel: Improving and Managing Kubernetes at Scale