Co-Location of CPU and GPU Workloads with High Resource Efficiency
summary
Describes co-locating long-running CPU services and AI batch GPU jobs on Kubernetes to increase resource utilization. Covers creating a new QoS class derived from BestEffort, introducing node-level cgroups for batch jobs, using a PodGroup CRD for gang scheduling, and measuring utilization on a production cluster of 100+ NVIDIA Tesla P100 GPU nodes and 500+ CPU nodes to achieve a ~10% utilization gain.
AI-written summary · about
chapters (13)
- 0:00Intro
- 0:59Cluster Scale
- 1:46Should We Care About Utilization?
- 2:37Why Low Utilization?
- 7:10Workloads - How to increase utilization?
- 8:36Workloads - Put them together
- 14:49Current Features in Kubernetes
- 16:18Problems - QoS Class
- 22:34Build Colocation with Native Feature - Kubelet
- 25:46Colocation on Kubernetes - Resource Agent
- 26:33More related works
- 28:15Architecture
- 29:38Results - Services + Jobs
related talks
37mCo-Location of CPU and GPU Workloads with High Resource Efficiency
43mThe GPU Utilization Problem: What’s Going Wrong and How to Solve It
36mProduction GPU Cluster with K8s for AI and DL Workloads
38mImproving GPU Utilization using Kubernetes
33mAI workloads on Kubernetes - How to maximize GPU utilization and cut costs
38mKubernetes for AI Workloads – What Works and What… Doesn’t
22mBeyond Single-Cluster Limits: Scaling GPU Workloads Across Kubernetes With…
28mGPU Reservations: Maximizing Utilization and Fairness Across Teams