vLLM: Easy, Fast, and Cheap LLM Serving for Everyone
summary
vLLM claims a 10x speedup over Hugging Face and 2–3x vs TGI by reorganizing KV cache into block-based page attention, enabling much higher GPU memory utilization; the repo has ~49k stars and vLLM powers a Prime Day load of ~3 million tokens/min across ~80,000 inference chips. It ships multi‑hardware support, weight/KV/activation quantization, persistent batching and multiprocessing to cut CPU overhead (old profile: 33% server, 29% prep, 38% model), and a W1 architecture update that roughly doubles common-model throughput.
AI-written summary · about
chapters (12)
- 0:00vLLM introduction and mission
- 1:02Page attention and KV cache
- 2:09Simple API and compatibility
- 3:14Hardware and ecosystem support
- 4:40Real-world deployments and PyTorch
- 5:44Parallelism strategies for scaling
- 6:46Advanced inference optimizations
- 7:50Quantization and compression
- 8:54Reducing CPU overhead
- 10:59Performance versus usability
- 12:26Community growth and PyTorch Foundation
- 13:52Ecosystem collaboration and vision
related talks
25mvLLM: Easy, Fast, and Cheap LLM Serving for Everyone
35mThe State of vLLM
38mOptimizing vLLM Performance through Quantization
21mScaling LLM Inference with vLLM
28mDatabricks' vLLM Optimization for Cost-Effective LLM Inference
24mLLM Inference Optimization From Simple To Sophisticated
39mAccelerating Open-Source RL and Agentic Inference with vLLM
40mScalable and Efficient LLM Serving With the VLLM Production Stack