Scalable Solutions for Running Large Language Models
summary
Adaptive autoscaler chooses between collocation (single replica handles prefill+decode) and segmentation (separate prepare and decode replicas) using measured decision points D1/D2 to optimize throughput and latency, switching when segmentation becomes more efficient. Speculative decoding plus online fine‑tuning yields 2.3x speedup versus baseline LLM inference; sticky routing with an L1/L2 prompt cache reduces repeated computation by routing users to the same GPU; platform-level optimizations (CUDA/ROCm GPU kernels) deliver 40% cost savings on average.
AI-written summary · about
chapters (7)
- 0:00Introduction to autoscaling for large language models
- 2:14Faster LLM inference
- 5:07Sticky routing and prompt caching
- 7:39Autoscaling LLM workloads
- 10:32Collocation vs segmentation
- 12:48Cost and performance results
- 13:53Summary and future outlook
related talks
20mSGLang: An Efficient Open-Source Framework for Large-Scale LLM Serving
15mDistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language...
15mResource Multiplexing in Tuning and Serving Large Language Models
24mLLM Inference Optimization From Simple To Sophisticated
19mSystem for Scaling Large Language Models
16mWLB-LLM: Workload-Balanced 4D Parallelism for Large Language Model Training
26mEfficiently serving LLMs at scale.
31mDeploying Large Language Models with Ease