LLM Inference Arithmetics
summary
Explains how parameter counts map to VRAM and compute so numbers mean something: Gemma 2B is closer to a 3B as a parameter-vs-storage example, a 7B model converts to roughly ~14GB plus 20% for activations and overhead, training demands ~three times the memory of inference so inference is ~1/3 cost of training for typical setups, and token-by-token decoding is quadratic in sequence length but performance improves because KV-Cache stores keys and values so generation trades extra memory for much less repeated compute.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 2:32Key takeaways
- 5:43Neural network basics
- 8:31Training vs inference
- 15:40Transformer overview
- 22:42Attention computation
- 26:57KV cache impact
- 30:44Self-hosting models
- 33:08Conclusion
related talks
30mLLM Inference Arithmetics: the Theory behind Model Serving
34mOptimizing AI Inference with ML Compilers & Hardware
30mUnderstanding the end-to-end LLM training and inference pipeline
24mLLM Inference Optimization From Simple To Sophisticated
38mOptimizing LLM Inference for the Rest of Us
30mAccelerated LLM Inference with Anyscale
41mSub-Linear Algorithms meets Large Language Models
10mGemma 3 architecture and design