Running AI Workloads in Containers and Kubernetes
summary
Containers inject host GPU user space libraries at runtime using CDI specs so Docker 25 and runtimes can expose GPUs without bundling drivers and Nvidia ctk generates those specs. Kubernetes uses a device plugin plus a GPU operator to install drivers and advertise resources. Sharing options are explicit: full GPU time‑slicing, MPS space partitioning, or hardware partitions via MIG with memory and latency tradeoffs. New Dynamic Resource Allocation (alpha since 1.26) adds device classes, namespaced resource claims, constraints (for example bind a NIC VF and GPU by pci_root), and a fake‑GPU demo runnable on a laptop.
AI-written summary · about
chapters (7)
- 0:00Introduction to containers and GPUs
- 3:09Challenges with GPU containers
- 8:43Container Device Interface (CDI) explained
- 16:45GPU sharing in Kubernetes
- 21:23Dynamic Resource Allocation in Kubernetes
- 25:58Future of GPU support in Kubernetes
- 27:21Q&A
related talks
32mScaling AI Workloads with Kubernetes: Sharing GPU Resources Across Multiple Containers- 32mScaling AI Workloads with Kubernetes: Sharing GPU Resources Across Multiple Containers
33mAI workloads on Kubernetes - How to maximize GPU utilization and cut costs
39m✨ GPU in My Containers!
30mExplain How Kubernetes Works With GPU Like I’m 5
26mState of the GPU(nion)
32mScaling Kubernetes Clusters for Generative Models: Managing GPU Resources for AI App...
38mImproving GPU Utilization using Kubernetes