Ray in Production: Engineering Lessons from Spotify's Hendrix Framework
summary
Spotify’s Hendrix platform runs Ray on Kubernetes to provision on-demand GPU infrastructure, orchestrate >1 million GPU hours/month, and serve >10,000 batch LLM jobs/month. It uses Ray Data for tokenization and packing and Ray Train with FSDP2/HSDP tuned to cluster topology, exposes a three-layer config (cluster, experiment, training recipes) with overridable defaults, and adopts heterogeneous runtime sidecars for specialized inference and feature workloads. Engineering lessons include topology-aware autoscaling and troubleshooting an NCCL deadlock caused by uneven end-of-epoch behavior across workers.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 1:22Platform overview
- 2:24Usage trends
- 2:55Workflows and configuration
- 6:37Infrastructure setup
- 9:20Scaling with Kueue
- 11:17Orchestration patterns
- 18:40Elasticity and scaling
- 20:45Q&A session
related talks
30mSpotify Harnesses Ray for Next-Gen AI Infrastructure
26mIntroduction to Distributed ML Workloads with Ray on Kubernetes
28mRay on Public-Cloud k8s - experiments, lessons & best practices
31mNext-Generation Data Science Workflows Using Ray
28mScaling AI & Machine Learning Workloads With Ray on AWS, Kubernetes, & BERT
21mLeveraging Ray to Enhance Machine Learning Models in Finance
29mAdopting Ray @ LinkedIn
22mRay, a Unified Distributed Framework for the Modern AI Stack