vLLM & Ray Updates
summary
vLLM reimplements its GPU runner as Model Runner V2, splitting a 6,000‑line file into several ~1,000‑line modules, decoupling per‑request state, moving input prep onto GPU and adopting an async‑first design for up to 50% performance improvement at high concurrency on small models with large GPUs. vLLM adds vLLM Omni for autoregressive plus diffusion multimodal IO, wide expert parallelism on H200/GB200 for multi‑node MOE to maximize KV cache, RDMA disaggregated serving and a new vLLM IR. Ray reports native RDMA support, 7x yearly download growth, and NVIDIA’s NeMo Curator hits 99.5% GPU utilization.
AI-written summary · about
chapters (11)
- 0:00vLLM introduction
- 1:07Model Runner V2 improvements
- 2:12vLLM Omni input/output
- 2:55Wide expert parallelism
- 3:36Q2 roadmap updates
- 4:38Ray introduction
- 5:39Ray for AI scaling
- 6:24Ray actors for RL
- 7:48Ray in LLM reinforcement
- 8:29AI data pipelines
- 9:57Ray ecosystem future
related talks
31mThe Evolution of Multi-GPU Inference in vLLM
15mvLLM: Easy, Fast, and Cheap LLM Serving for Everyone
28mServing LLMs for Agents at Scale with Ray + vLLM
21mScaling LLM Inference with vLLM
25mvLLM: Easy, Fast, and Cheap LLM Serving for Everyone
35mThe State of vLLM
28mScaling Multimodal Data and Reinforcement Learning with Ray
39mAccelerating Open-Source RL and Agentic Inference with vLLM