Scaling RL with Ray, Kubernetes, and TPUs
summary
Reference architecture using Ray + GKE + TPUs with concrete numbers and tools: Ray adds sub-second cold starts via native gVisor sandboxes and promises substrate wake <100ms to eliminate boot time; tested at >100,000 concurrent sandboxes (200k on one cluster, million across clusters). TPUs become first-class in Ray 2.55 with prebuilt images and dashboard metrics. Time-slicing reduces GPU count by ~66% and pushes utilization toward 95%, while a passive standby head node cuts recovery from minutes to seconds.
AI-written summary · about
chapters (12)
- 0:00Introduction to RL scaling
- 1:10RL stack and Ray orchestration
- 4:42Hardware and TPU support
- 8:12Sky RL and framework portability
- 10:01Head node reliability improvements
- 11:06Punam on infrastructure inefficiencies
- 13:18Time slicing for accelerator utilization
- 23:37RL scheduler and batching
- 25:48Sandboxing at scale
- 27:13Observability and debugging
- 28:38Open RL multi-tenant service
- 31:50Future roadmap and next steps
related talks
28mScaling Multimodal Data and Reinforcement Learning with Ray
28mRay on Public-Cloud k8s - experiments, lessons & best practices
28mScaling AI & Machine Learning Workloads With Ray on AWS, Kubernetes, & BERT
26mIntroduction to Distributed ML Workloads with Ray on Kubernetes
22mRay, a Unified Distributed Framework for the Modern AI Stack
35mScaling Ray Train to 10K Kubernetes Nodes on GKE
22mLarge Scale Data Loading and Data Preprocessing with Ray
31mNext-Generation Data Science Workflows Using Ray