RollPacker: Taming Long-Tail Rollouts for RL Post-Training with Tail Batching
summary
RollPacker implements tail batching: it groups prompts that produce very long responses into occasional long rounds and uses speculative execution to run extra prompts so most rounds are short and balanced, preserving on‑policy semantics by eventually including every prompt. System components include an adaptive parallelism planner, a reward scheduler that collocates judge models, and a stream trainer that reclaims rollout GPUs for training. Deployed on up to 128 H800 GPUs, it speeds end‑to‑end RL post‑training 2.03–2.56× over veRL and up to 2.24× vs RLHFuse; code is on GitHub.
AI-written summary · about
chapters (15)
- 0:00RollPacker and RL post-training
- 2:29Rollout bottlenecks in RL
- 3:34Tail batching for efficiency
- 5:21Long prompt queue and semantics
- 6:25Tail batching impact on accuracy
- 7:30Efficiency gains and response length
- 8:34Overhead and KV cache pressure
- 10:01RollPacker system overview
- 11:07Parallelism planner for TP selection
- 11:49Reward scheduler and GPU collocation
- 12:54Stream trainer for GPU reassignment
- 13:57Evaluation setup and metrics
- 14:41Performance analysis across models
- 15:45Step breakdown and scalability
- 16:26Conclusion and open-source
related talks
15mRollArt: Disaggregated Multi-Task Agentic RL Training at Scale
20mEfficient Reinforcement Learning
19mRL Environments at Scale
19mDynaRL: Flexible and Dynamic Scheduling of Large-Scale Reinforcement Learning Training
32mEnd-to-End Optimizing Multi-Turn RL and High-Performance Inference in Agents with...
36mSkyRL: Democratizing RL Training for Researchers
21mWhen Prompting Runs Out: Post-Training an Open Model
23mCost-Effective RL Post-Training with Ray and AWS Trainium