Cost-Effective Scheduling of a Massive Number of Containers in Kubernetes
summary
JD describes a production scheduler called Archimedes that handles a massive Kubernetes fleet (about 5,000 servers and ~225,000 pods) and drove the platform through JD’s June 18 sale with $24.7 billion in transactions. Engineering combines second‑level monitoring (ClickHouse time‑series), a ContainerFS layer, ML CPU/memory sizing and co‑scheduling policies, and rack‑aware placement to cut scheduling latency ~90% and lift cluster utilization from ~40% to ~61%.
AI-written summary · about
chapters (20)
- 0:00Intro
- 1:07JD Retail Technological Infrastructure Group
- 3:38JDOS (JD Data Center OS) Architecture
- 4:59Resource Management: Challenges
- 7:46A Typical JDOS Cluster
- 8:09Bottleneck Analysis and Optimization
- 10:36Customization and Optimization
- 11:41Preemption and Admission Control
- 12:56Group Scheduling and Binding
- 17:47A Closed Loop System
- 18:06Advanced Scheduling
- 20:50Right Sizing of CPU Resource
- 24:05Resource Usage and Performance Tradeoff
- 26:05Memory Scheduling for Online Services
- 27:45Memory Utilization of Online Services
- 28:27Host Selection Optimization
- 30:12Mixed Workload Placement
- 30:40Hybrid Resource Pools
- 32:59June 18 Anniversary Sale (2019)
- 34:36Acknowledgements
related talks
38mCost-Effective Scheduling of a Massive Number of Containers in Kubernetes
40mLarge-Scale K8s Cluster Operation and Management
11mHow Jenga & Tetris Can Help Run More Containers with Lesser Costs...
24mKubernetes Batch Processing at Scale - A Scheduling Perspective
35mBuilding Armada – Running Batch Jobs at Massive Scale on Kubernetes
15mDemo: Resource(Cost) optimization on planet scale Kubernetes clusters.
36mRight-Sizing and Auto-Scaling of MySQL Containers in Kubernetes
47mBuilding sustainable architectures with Kubernetes