Optimizing LLM Inference for the Rest of Us
summary
Kubernetes workflows for LLM inference focus on concrete fixes: separate prefill vs decode to match CPU-bound and memory-bound phases, measure time to first token and time-per-token, and use KV caching and Kubernetes 1.35’s container-as-volume plus a cluster cache (Spiggel) to shave minutes off cold start. Quantization matters — Deepseek v1 needs ~1.37 TB at full precision so an A100 (≈80GB) can’t hold it. Use dynamic resource claims for GPUs and the endpoint-picker extension to route by model_id; LLMD provides reproducible patterns and benchmarks.
AI-written summary · about
chapters (17)
- 0:00Optimizing LLM inference locally
- 1:32Why large language models are getting bigger
- 4:17LLM inference concepts
- 7:51Prefill and decode pipeline
- 10:18Kubernetes for local models
- 14:35Kubernetes for LLM workloads
- 17:04Optimizing container sizes
- 19:28Memory needs for models
- 20:10Caching models with Spiggel
- 21:12Model storage with volumes
- 22:15Dynamic GPU allocation
- 24:43Solving resource scarcity
- 27:54Network challenges
- 31:25Routing by model ID
- 34:54LLMD project patterns
- 36:59LLM optimization recap
- 37:42Q&A
related talks
42mStop Treating LLMs Like REST APIs
33mllm-d: Kubernetes Native Distributed Inferencing
28mProduction-Ready LLMs on Kubernetes: Patterns, Pitfalls, and Performa...
32mScaling Generative AI Inference with llm-d
24mLLM Inference Optimization From Simple To Sophisticated
27mServing PyTorch LLMs at Scale: Disaggregated Inference With Kubernetes and Llm-d
30mAccelerated LLM Inference with Anyscale
22mIntroduction to Large Language Models (LLM) on Kubernetes