Scaling Kubernetes Clusters for Generative Models: Managing GPU Resources for AI App...
summary
Kubernetes uses the NVIDIA GPU Operator plus the device plugin to label nodes, enable a Mig manager, and expose GPU resources; MIG physically shards cards into up to 7 compute / 8 memory slices (A100 40GB → 7×5GB slices) while Kubernetes time-slicing exposes logical replicas (replica count × GPU count) for round‑robin access. MIG gives memory and compute QoS and requires node reboot to reconfigure; time‑slice is dynamic but risks cross‑pod resets and no memory isolation. Also recommended: FP16 inference, FlashAttention, speculative decoding and distillation.
AI-written summary · about
chapters (13)
- 0:00Scaling AI workloads with Kubernetes
- 1:04GPUs essential for AI workloads
- 2:29Kubernetes for GPU management
- 3:10Multi-instance GPU (MIG) partitioning
- 9:53Compute instances and MIG isolation
- 11:14NVIDIA GPU Operator setup
- 13:43GPU node management best practices
- 15:28Sharing GPU resources
- 17:13MIG profiles in Kubernetes
- 19:43Time slicing vs MIG partitioning
- 23:57Optimizing deep learning workloads
- 27:08GPU resource management summary
- 27:52Q&A on GPU partitioning
related talks
32mScaling AI Workloads with Kubernetes: Sharing GPU Resources Across Multiple Containers- 32mScaling AI Workloads with Kubernetes: Sharing GPU Resources Across Multiple Containers
33mAI workloads on Kubernetes - How to maximize GPU utilization and cut costs
38mImproving GPU Utilization using Kubernetes
30mRunning AI Workloads in Containers and Kubernetes
30mGPU Sharing at CERN: Cutting the Cake Without Losing a Slice!
31mKubernetes For GPU Powered Machine Learning Workloads In Academia
32mA Deep Dive on Supporting Multi-Instance GPUs in Containers and Kubernetes