Scaling Ray Train to 10K Kubernetes Nodes on GKE
summary
Google runs Ray Data and Ray Train across a 10,000-node GKE cluster by sharding Landsat‑8 (≈3.3 PB) into about 22,000 Ray jobs, using KubeRay plus a Kubernetes-native queue to admit only ~1,000 ephemeral Ray clusters at a time so failures retry cheaply. Tests use 10,000 e2‑standard‑4 nodes (≈40k CPUs, 160 TB RAM), bake deps into images, increase reconcile concurrency, and rely on the GCS Fuse CSI driver with metadata caching and parallel prefetch to SSDs — the setup succeeds and completes demo epochs with graceful restarts.
AI-written summary · about
chapters (8)
- 0:00Ray on GKE introduction
- 2:23Scalability dimensions
- 4:53Benchmark testing strategy
- 11:03Capacity management with Kueue
- 14:43GCS Fuse CSI integration
- 19:11End-to-end training demo
- 26:11Getting started resources
- 27:04Q&A session
related talks
28mRay on Public-Cloud k8s - experiments, lessons & best practices
19mScaling Frontier AI with Ray on Kubernetes
16mScaling Massive Transformer Training with KubeRay
31mHosting AI models with Ray and GKE
39mSailing Ray Workloads with KubeRay and Kueue in Kubernetes
28mScaling AI & Machine Learning Workloads With Ray on AWS, Kubernetes, & BERT
26mIntroduction to Distributed ML Workloads with Ray on Kubernetes
34mScaling RL with Ray, Kubernetes, and TPUs