Building & Operating a Large-scale HPC AI Cluster on Kubernetes
Talkat KubeCon + CloudNativeCon Europe 2025 · on Thursday, 2025-04-03· at 16:45· Level 1 | Hall Entrance S10 | Room B·
summary
Shares how a cloud research cluster with 12K GPUs runs Slurm on Kubernetes by keeping slurm daemon on the host, containerizing control-plane components, and using a custom CLI that creates per-user login pods via Helm. Implements an automount daemonset to add NFS filesystems without restarting jobs, extensive telemetry and node health checks to isolate bad GPUs by ID, and rolling updates that drain Slurm nodes so long-running training stays reliable; operation proves feasible though nontrivial.
AI-written summary · about
chapters (16)
- 0:00Building AI HPC on Kubernetes
- 1:48ML training overview
- 4:13ML research lifecycle challenges
- 6:21Researcher experience design
- 8:27Distributed training reliability
- 11:47On-premises to cloud clusters
- 15:04Kubernetes choice
- 17:14Kubernetes cluster architecture
- 19:23Researcher workflows on Kubernetes
- 21:50Security vs resource isolation
- 22:32Kubernetes infrastructure management
- 24:39Deployment simplification
- 25:03Node configuration consistency
- 26:50Storage for AI HPC workloads
- 28:36Migration key takeaways
- 30:41Q&A on reliability
related talks
40mBuilding Custom GPU Clusters at Scale: Using Kubespray To Create High-Performance AI Infrastructure
24mCluster Management for Large Scale AI and GPUs: Challenges and Opportunities
36mProduction GPU Cluster with K8s for AI and DL Workloads
38mKubernetes for AI Workloads – What Works and What… Doesn’t
27mHarnessing Kubernetes for Scalable AI/ML Workloads
24mDynamic Big Data in the Cloud with Kubernetes
05mHow To Run Kubernetes Pods On My Slurm-Based HPC Center
31mKubernetes For GPU Powered Machine Learning Workloads In Academia