How fast are LLM inference engines anyway?
summary
Benchmarks cover hundreds of runs across open models, engines (VLM, SGLang, Tensor TLM), and H100 GPUs with raw data and code posted. Key reproducible claims: first token <1s for many configurations, ≈1 RPS for Quen3 on 128 in / 1024 out under generation-heavy loads, and ≈4 RPS on VLM for 1024 in / 128 out where prefill dominates. BF16 runs are slower than FP4/FP8 prospects, SG Lang had 8bit quants buggy, and throughput is measured per GPU with a methodology that sweeps single-request latency to max-concurrency RPS.
AI-written summary · about
chapters (4)
- 0:00Introduction to LLM inference
- 5:05Benchmarking open model engines
- 8:38Performance results and insights
- 14:19Q&A and closing remarks
related talks
24mLLM Inference Optimization From Simple To Sophisticated
51mPractical LLM Inference in Modern Java
50mKa-Ching! How Enterprises Measure LLM Performance and Cost
21mScaling LLM Inference with vLLM
13mHow Open-Source vLLM Topped the Artificial Analysis Leaderboard
47mPractical LLM Inference in Modern Java
28mDatabricks' vLLM Optimization for Cost-Effective LLM Inference
31mFrom ML to LLM on device AI in the browser