Spotify Harnesses Ray for Next-Gen AI Infrastructure
summary
Spotify runs Ray on GKE and claims support for LLM training beyond 70B parameters, growing Hendrix from 80 to 120 teams and from ~100 to ~1,400 Ray clusters/week while scaling Kubernetes from ~500 to ~4,500 nodes. The platform adds a server-side Kubernetes webhook to centralize policy, uses Q with a K plus Google DWS for gang scheduling and fair GPU allocation, supports multi and sidecar worker groups for mixed CPU/GPU stages, and mounts checkpoints with GCS fuse; these changes improve readiness, sharing, and cost control.
AI-written summary · about
chapters (22)
- 0:00Introduction
- 0:41Spotify ML Platform
- 1:33Spotify HRX Growth
- 1:57Themes
- 2:40Architecture
- 3:27Evolution
- 5:01Network bandwidth
- 6:01Platform logic migration
- 7:36Q with a K
- 9:41User Experience
- 11:03User Facing Changes
- 12:18Features
- 12:43Backstage
- 13:40Config Layer
- 14:30Array Cluster
- 15:07Ray Job
- 16:03Multiple Worker Groups
- 17:14Sidecar Worker Group
- 18:52Storage Optimization
- 20:44QDWS
- 22:30Connecting all the dots
- 25:21Wrap up
related talks
32mRay in Production: Engineering Lessons from Spotify's Hendrix Framework
28mScaling AI & Machine Learning Workloads With Ray on AWS, Kubernetes, & BERT
19mScaling Frontier AI with Ray on Kubernetes
26mIntroduction to Distributed ML Workloads with Ray on Kubernetes
28mRay on Public-Cloud k8s - experiments, lessons & best practices
28mScaling Multimodal Data and Reinforcement Learning with Ray
44mPanel: ML Platform on Ray
31mNext-Generation Data Science Workflows Using Ray