Spotify's Approach to Distributed LLM Training with Ray on GKE
summary
Spotify trains LLMs larger than 70B parameters using Ray on GKE with H100 GPUs, compact placement and NCCL Fast Socket to cut inter‑GPU latency, and uses ephemeral Ray clusters plus queuing (Kubernetes Q) and Dynamic Workload Scheduling to densify scarce GPU capacity. Container image preloading yields up to 29x cold‑start improvements; GCS is mounted via GCS Fuse for checkpointing so long‑running jobs resume after interruptions. Platform enforces whole‑GPU allocation today and reports these measures materially improve utilization and reduce wasted reserved capacity.
AI-written summary · about
chapters (9)
- 0:00Introduction and challenges
- 2:23Platform orchestration
- 3:01GKE features for AI
- 14:20Spotify platform overview
- 14:43Platform abstraction layer
- 17:15GPU resource optimization
- 18:14GKE optimization strategies
- 25:00Scheduling and concluding
- 28:07Q&A session
related talks
26mIntroduction to Distributed ML Workloads with Ray on Kubernetes
18mServing Fast and Smart: Getting the Most Out of Ray Serve LLM on GKE
28mRay on Public-Cloud k8s - experiments, lessons & best practices
28mScaling LLM Workloads on TPUs with Ray: From Pre-Training to Inference
28mEfficient Deep Learning Training with Ludwig AutoML, Ray, and Nodeless Kubernetes
28mServing LLMs for Agents at Scale with Ray + vLLM
28mScaling AI & Machine Learning Workloads With Ray on AWS, Kubernetes, & BERT
19mScaling Frontier AI with Ray on Kubernetes