Right Resource, Right Job: Heterogeneous Ray Clusters on Amazon EKS
summary
Robinhood moves a homogeneous RayCluster on Amazon EKS to a heterogeneous setup using KubeRay, Carpenter node pools, Ray Data and Ray Train, and custom resources that pin tasks to worker groups. A CPU-heavy preprocessing worker group (example GPU small: 8 CPU 32GB 1 T4) autoscale independently from GPU worker groups, reducing GPU idle time and contention and allowing scale-to-zero. Results show 67% cost reduction and 4x runtime improvement on a large training workload.
AI-written summary · about
chapters (17)
- 0:00Introduction to talk
- 1:33Robin Hood ML platform overview
- 4:22Ray adoption journey at Robin Hood
- 6:27King's Cross platform architecture
- 9:17Challenges with homogeneous clusters
- 12:32Observability and resource profiling
- 13:37Worker groups and node pools
- 16:03Resource profiles for ML workloads
- 18:33Heterogeneous cluster setup
- 20:22Resource-aware scheduling in Ray
- 21:48Elastic hardware scaling
- 22:50Ray Data and Ray Train integration
- 24:15Heterogeneous job configuration
- 25:39Results and cost savings
- 27:07Lessons learned from journey
- 29:16Future plans and next steps
- 30:01Q&A session
related talks
39mSailing Ray Workloads with KubeRay and Kueue in Kubernetes
28mRay on Public-Cloud k8s - experiments, lessons & best practices
24mSkyRay: Seamlessly Extending KubeRay to Multi-Cluster Multi-Cloud Operation
09mLightning Talk: Optimize the Ray Schedule and Autoscaling on Kubernetes Cloud: A Hetero...
26mIntroduction to Distributed ML Workloads with Ray on Kubernetes
25mHow to take your Kubernetes cluster to next level with Karpenter, Ray and Amazon EKS?
31mRay at Scale: Apple's Approach to Elastic GPU Management
35mScaling Ray Train to 10K Kubernetes Nodes on GKE