RollArt: Disaggregated Multi-Task Agentic RL Training at Scale
Talkat 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI '26) · 2026-07-13..07-15·
summary
RollArt disaggregates multi-task agentic RL by mapping prefill tasks to compute-optimized GPUs, decode tasks to bandwidth-optimized GPUs, and environments to CPU clusters, decoupling rollout at the trajectory level and offloading stateless reward scoring to serverless; it overlaps rollout and training with staleness-bounded asynchronous weight sync and runs on H800/H20 clusters. Evaluation reports 1.31–2.05× training time reduction across baselines and a production run scaling a hundreds-of-billions-parameter MoE model on over 3,000 GPUs, demonstrating improved throughput and stability.
AI-written summary · about
chapters (15)
- 0:00RollArt and agentic RL
- 1:01Heterogeneous workflows
- 2:27Disaggregated infrastructure
- 3:30Hardware affinity
- 4:11Long tail execution
- 4:53Rewards as stateless functions
- 5:58RollArt’s three-plane architecture
- 7:03Resource and data plane setup
- 8:06Worker-level programming interface
- 9:12Cluster execution and mapping
- 10:15Trajectory orchestration
- 11:45Cross-class optimizations
- 12:09Evaluation setup
- 13:10Production deployment
- 14:35Conclusion
related talks
17mRollPacker: Taming Long-Tail Rollouts for RL Post-Training with Tail Batching
15mRLBoost: Harvesting Preemptible Cloud Resources for Cost-Efficient Reinforcement Learning
19mRL Environments at Scale
21mTraining AI for Space:
23mCost-Effective RL Post-Training with Ray and AWS Trainium
26mMSRL: Distributed Reinforcement Learning with Dataflow Fragments
20mEfficient Reinforcement Learning
27mLessons from Unifying Simulation and RL Training