Co-Location of CPU and GPU Workloads with High Resource Efficiency
summary
Describes co-locating long-running services and AI batch jobs on the same Kubernetes nodes (mixing CPU and NVIDIA Tesla P100 GPU machines) to raise resource utilization. Covers concrete Kubernetes changes and mechanisms: creating a new QoS class derived from BestEffort, adding node-level cgroups for batch jobs, using a PodGroup CRD to enable gang scheduling, and how utilization was measured in a cluster of 100+ GPU and 500+ CPU nodes achieving a ~10% utilization increase.
AI-written summary · about
chapters (13)
- 0:00Start
- 0:59Cluster Scale
- 1:46Should We Care About Utilization?
- 2:37Why Low Utilization?
- 4:10Increasing Utilization Brings Significant Cost Saving
- 7:10Workloads - How to increase utilization?
- 8:36Workloads - Put them together
- 14:49Current Features in Kubernetes
- 16:18Problems - QoS Class
- 22:34Build Colocation with Native Feature - Kubelet
- 25:46Colocation on Kubernetes - Resource Agent
- 29:38Results - Services + Jobs
- 29:56OpenKruise - Automate everything!
related talks
37mCo-Location of CPU and GPU Workloads with High Resource Efficiency
38mImproving GPU Utilization using Kubernetes
43mThe GPU Utilization Problem: What’s Going Wrong and How to Solve It
33mAI workloads on Kubernetes - How to maximize GPU utilization and cut costs
22mBeyond Single-Cluster Limits: Scaling GPU Workloads Across Kubernetes With…
36mProduction GPU Cluster with K8s for AI and DL Workloads
32mScaling AI Workloads with Kubernetes: Sharing GPU Resources Across Multiple Containers
37mBuilding Distributed TensorFlow Using Both GPU and CPU on Kubernetes [I]