Adapting Ray to Scale RL Training for Nemotron
summary
NVIDIA adapts Ray to run Nemotron RL at 3K+ GPU scale by changing Ray configs, reducing actor count 42%, and moving heavy GCS work into separate IO contexts (available in Ray 2.55+), cutting cluster init from 33 minutes to under 3. Topology-aware actor placement (NUMA and NVLink domain binding) raises throughput ~10–20%. A CPU-backed transfer queue plus RDMA and GPU Direct RDMA speeds rollout and distillation data paths 2–4x. Head node is pinned to orchestration and GCS thread pool and registration timeouts are increased for resilience.
AI-written summary · about
chapters (7)
- 0:00Introduction to Nemotron
- 2:14Nemotron model architecture
- 7:56Challenges in RL training
- 9:20Ray architecture primer
- 10:45Resiliency tuning in Ray
- 17:50Efficiency optimizations with Ray
- 24:54Key takeaways and Q&A
related talks
22mLarge Scale Data Loading and Data Preprocessing with Ray
23mImproving Ray for Large-scale Applications
28mScaling Multimodal Data and Reinforcement Learning with Ray
37mEvolving Ray Core: Scalability, Reliability, and Compiled Graphs
28mOptimizing Large-Scale Model Training with Ray Compiled Graphs
21mHow Ray and Anyscale Make it Easy to do Massive-scale ML on Aerial Imagery
15mRay: A System for Distributed Applications
31mFaster Model Serving with Ray and Anyscale