Verl: A Flexible and Efficient RL Framework for LLMs
summary
verl is an open source RL pipeline that combines a hybrid-controller (central logic + many SPMD controllers) with WorkerGroup and ResourcePool for cluster orchestration, claiming production use and 14K+ stars, 2K forks and ~400 contributors. It implements a token-in token-out agent loop and an async server mode to drop generation bubbles and raise GPU/CPU utilization. Integrations include FSDP/FSDP2, Megatron-LM, vLLM and SGLang, supports PPO/GRPO/DAPO, and shows a working DeepSeek-671B recipe running on 96 H100 GPUs.
AI-written summary · about
chapters (6)
- 0:00Importance of large-scale RL
- 1:34System challenges in RL
- 2:42Verl framework design
- 5:27Agentic RL and async updates
- 12:08Roadmap and community
- 20:41Q&A session
related talks
24mverl: An Open Source Large Scale LLM RL Framework for Agentic Tasks
39mAccelerating Open-Source RL and Agentic Inference with vLLM
17mRLinf: Flexible and Efficient Large-Scale Reinforcement Learning...
27mSkyRL: A Modular RL Library for LLM Agents
14mRLlib: Scalable RL for TensorFlow, PyTorch, and Beyond
35mThe State of vLLM
32mByteDance's Platform for Reinforcement Learning from Human Feedback
19mRL Environments at Scale