From Responses To Trajectories: Multi-Turn and Multi-Environ...
summary
Claims that post-training moves from single-response optimization to GRPO on trajectories where the training unit is multi-turn rollouts, not completions. Demonstrates TRL’s runtime primitives: TRL environment factory, masking to optimize only model tokens, rollout batching, and client-server OpenEnv integration. Shows multimodal, multi-environment runs (Carla driving simulator) where a QN3 0.6B model learns lane changes and emergency stops in about 50 steps using remote HF Spaces and vLLM bottleneck considerations for scalable generation. Includes a fallback rollout function for full-generation control and examples for LLMs and BLMs.
AI-written summary · about
chapters (10)
- 0:00Trajectory-based RL post-training
- 2:48Trajectories need environments
- 4:54GRPO to multi-turn rollouts
- 7:44TRL runtime loop
- 8:49Masking tokens in sequences
- 9:54Scaling to multiple environments
- 13:23Multimodal environments like browsers
- 16:58Autonomous driving with Carla
- 20:28Capabilities and roadmap
- 22:57Q&A
related talks
19mTraining Agentic Reasoners
32mEnd-to-End Optimizing Multi-Turn RL and High-Performance Inference in Agents with...
36mSkyRL: Democratizing RL Training for Researchers
27mScaling an Open Environments Ecosystem for Reinforcement Learning
21mWhen Prompting Runs Out: Post-Training an Open Model
42mPost-Training Small Language Models: the adventures of a practitioner
17mRollPacker: Taming Long-Tail Rollouts for RL Post-Training with Tail Batching
19mRL Environments at Scale