SkyRL: Democratizing RL Training for Researchers
summary
SkyRL presents a modular RL stack that ties Megatron training, vLLM serving, and Ray orchestration into fully async RL pipelines that run 350B+ MoE and trillion-parameter models with 256k contexts. It exposes a multi-tenant Tinker Engine so teams run Tinker-compatible scripts on shared GPUs; Trajectory AI reports 10,000+ runs and Laya runs SFT on 500+ H100s. Benchmarks claim sticky least loaded routing and 1.4–4x generation throughput, 70% relative reward uplift (16%→27%) on Apex agents, FP8 speedups ~20–23%, and a 7.53s sharded sync for a 1T model.
AI-written summary · about
chapters (14)
- 0:00Introduction to SkyRL
- 1:29What is SkyRL
- 2:31Key users and collaborators
- 5:22Post-training requirements in 2026
- 7:29Large scale post-training pipeline
- 9:38Supervised fine-tuning in SkyRL
- 11:23Agentic RL at scale
- 13:33Inference architecture
- 16:10Weight syncing improvements
- 19:47FP8 training and performance
- 22:37Customizability and Tinker API
- 24:26SkyRL Tinker Engine
- 26:53Roadmap and future plans
- 29:12Q&A session
related talks
27mSkyRL: A Modular RL Library for LLM Agents
19mTraining Agentic Reasoners
39mAccelerating Open-Source RL and Agentic Inference with vLLM
32mEnd-to-End Optimizing Multi-Turn RL and High-Performance Inference in Agents with...
44mWhy RL Environments Are All You Need for Building Agents
37mA practical perspective on training reinforcement learning bots.
39mRedefining LLM Training Through Atomic Components and Serverless TaaS
24mverl: An Open Source Large Scale LLM RL Framework for Agentic Tasks