AI Training & Inference on Kubernetes: Compute Patterns and Best Practices with Karpenter.
summary
Provides concrete patterns and knobs for running training and inference on Kubernetes using Karpenter: gang scheduling and placement-group use for multi-node GPU training, topology-aware node overlays and EFA-enabled inter-node tensor parallelism for large models, and GPU packing via MIG slices, MPS or time-slicing for small models. Shows how capacity reservations (ODCR, capacity blocks) and capacity buffers/headroom plus parallel image pulls and user-data model prefetching cut cold starts toward 1–2 minutes. Explains node-pool weights, spot fallback and static capacity behavior and where gaps remain.
AI-written summary · about
chapters (14)
- 0:00Introduction to AI on Kubernetes
- 3:16Inference workload patterns
- 5:49Cold start optimization for LLM inference
- 8:40Capacity buffers for inference
- 10:06GPU sharding and interconnects
- 14:04GPU time slicing and MIG
- 16:54GPU purchasing options
- 19:46Training workload patterns
- 21:33Static capacity for training
- 26:05Placement groups for training
- 29:18Gang scheduling for training
- 31:47Multi-workload cluster strategy
- 34:33Future improvements and resources
- 35:56Q&A and closing
related talks
16mAI + Kubernetes: What Beginners...
41mPanel: AI Infra Best Practices: Enterprise Do’s and Don’ts
28mTraining Large Language Models on Kubernetes
29mCutting Costs and Boosting Efficiency with Karpenter and KEDA on Kubernetes
32mScaling Kubernetes Clusters for Generative Models: Managing GPU Resources for AI App...
27mHarness the power of Karpenter to scale, optimize & upgrade K8s
33mAI workloads on Kubernetes - How to maximize GPU utilization and cut costs
25mScaling Machine Learning Models on Kubernetes