Training Large Language Models on Kubernetes
summary
Shows Kubernetes running multi-node LLM pretraining (Megatron/NVIDIA Nemo) on A100 DGX gear and compares bare metal vs K8s throughput, reporting <2% overhead when training GPT‑3 5B across 1/2/4 nodes (32 GPUs, InfiniBand). Provides ready scripts, data preprocessing and configs, and a scheduler that adds fractional GPUs via CUDA interception plus MIG/MPS orchestration, dynamic quotas and gang scheduling so distributed training scales near linearly and interactive/inference use cases coexist.
AI-written summary · about
chapters (11)
- 0:00Training LLMs on Kubernetes
- 1:03AI compute demand
- 3:55GPU access challenges
- 7:10Static GPU allocations
- 10:23Manual GPU sharing issues
- 11:28Why Kubernetes for AI
- 13:14AI scheduling solutions
- 15:42Run:AI’s platform
- 16:26LLM training projects
- 17:50Benchmarking LLM training
- 20:45Q&A on multicluster
related talks
31mKubernetes For GPU Powered Machine Learning Workloads In Academia
31mOvercoming Challenges in Serving Large Language Model
32mRunning and Fine-tuning Open Source LLMs on Google Kubernetes Engine
12mLightning Talk: Benchmarking Your Distributed ML Training on the K8s Platform
34mDemocratizing Machine Learning on Kubernetes [I]
24mEfficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM
21mMegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs
36mAccelerating High-Performance Machine Learning at Scale i...