The Evolution of Multi-GPU Inference in vLLM
summary
vLLM shards 405B-class models across GPUs using tensor + pipeline parallelism, rearchitects the control plane to use Ray compiled graph with GPU RDMA and Niel broadcast, and reports concrete wins: 900 GB fp16 model exceeds single-node 80 GB limits so it spans nodes; token latency targets ~10–20 ms; compiled graph yields up to 140x faster large-GPU transfers (20x without MV link) and 10–20% latency improvement for pipeline/2D setups.
AI-written summary · about
chapters (7)
- 0:00Intro
- 2:44Distributed Pretraining vs Inference
- 6:56Tensor Parallelism
- 10:51Pipeline Parallelism
- 15:35CPU bottleneck
- 22:56Results
- 24:51Questions
related talks
11mvLLM & Ray Updates
19mNo GPU Left Behind: Scaling Online LLM Training With...
15mvLLM: Easy, Fast, and Cheap LLM Serving for Everyone
21mScaling LLM Inference with vLLM
12mVertical Mobility: Inference from MVP to Trillion Parameters
27mServing PyTorch LLMs at Scale: Disaggregated Inference With Kubernetes and Llm-d
11mKeynote: Building a Large Model Inference Platform for Heterogeneous Chinese Chips Based on VLLM
34mHow To Evolve Your LLM Self-Hosting Platform: A Practical Guide To Adopting…