AI workloads on Kubernetes - How to maximize GPU utilization and cut costs
summary
NVIDIA GPUs are shared via three concrete methods: time slicing (supported everywhere but shows ~2x slower inference with seven pods due to context-switch overhead and no memory isolation), MIG (Ampere+ only, full isolation, A100 80GB smallest slice = 10GB) and MPS (parallel execution, configurable memory limits but requires hostIPC and runAsUser=1000). Kubernetes integration uses Device Plugins and GPU Operator; static partitions cause scheduling holes, so the NOS operator performs real-time partitioning for MIG/MPS, and Kubernetes 1.26 adds the dynamic resource allocation API for vendor drivers.
AI-written summary · about
chapters (8)
- 0:00Start
- 15:49Managing MIG Devices
- 18:23The issue with GPU sharing An example
- 20:29Dynamic GPU Partitioning (2)
- 20:46Dynamic GPU Partitioning with nos
- 24:47Dynamic Resource Allocation API (2)
- 25:44Key takeaways
- 26:58Question time!
related talks
32mScaling AI Workloads with Kubernetes: Sharing GPU Resources Across Multiple Containers- 32mScaling AI Workloads with Kubernetes: Sharing GPU Resources Across Multiple Containers
32mScaling Kubernetes Clusters for Generative Models: Managing GPU Resources for AI App...
38mImproving GPU Utilization using Kubernetes
30mRunning AI Workloads in Containers and Kubernetes
30mGPU Sharing at CERN: Cutting the Cake Without Losing a Slice!
26mState of the GPU(nion)
38mMinimizing GPU Cost for Your Deep Learning on Kubernetes