Scaling LLM Inference with vLLM
summary
vLLM provides multi‑accelerator inference with tensor parallelism, pipeline parallelism via Ray and an expert (Mixture of Experts) routing option, and supports NVIDIA, AMD and Google TPUs with a plug‑in model and 100k–300k downloads/day. LM Compressor implements state‑of‑the‑art quant schemes and a catalog of quantized models; a >1M evaluation sweep finds FP8 and int8 with tuned calibration recover 98–99% accuracy, int4 varies 95–99%. Quantization lets larger models fit constrained GPUs but latency vs QPS depends on format and dequantization costs.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 3:35vLLM as standard
- 7:08Scaling techniques
- 11:09Quantization lifecycle
- 13:58Not all quantized models are equal
- 16:24Deployment quantization
- 18:32Choosing quantization format
- 20:17Closing
related talks
38mOptimizing vLLM Performance through Quantization
35mThe State of vLLM
15mvLLM: Easy, Fast, and Cheap LLM Serving for Everyone
24mLLM Inference Optimization From Simple To Sophisticated
11mWhen Less Is More: A Technical Overview of LLMs and the Strength of Smaller Models
25mvLLM: Easy, Fast, and Cheap LLM Serving for Everyone
31mThe Evolution of Multi-GPU Inference in vLLM
39mAccelerating Open-Source RL and Agentic Inference with vLLM