Serving PyTorch LLMs at Scale: Disaggregated Inference With Kubernetes and Llm-d
summary
llm-d implements disaggregated prefill/decode on Kubernetes, running vLLM-backed model servers with a routing sidecar and NVIDIA NIXL for GPU-to-GPU KV transfers. Benchmarks on two H200 nodes compare a baseline of four TP=4 pods vs disaggregation with four prefill (TP=2) and two decode (TP=4) pods and show a clear per-GPU throughput win at mid concurrency. Best results appear for large models (70B+), long input/output sequences, and careful autoscaling.
AI-written summary · about
chapters (10)
- 0:00Introduction to LMD
- 0:44What is LMD architecture
- 2:30Why LMD stands out
- 4:41Valid passes for deployments
- 7:39PD disaggregation intro
- 9:45Disaggregated inference implementation
- 15:05Benchmarking PD disaggregation
- 18:42Best practices for PD disaggregation
- 20:08Key takeaways and resources
- 21:19Q&A
related talks
33mllm-d: Kubernetes Native Distributed Inferencing
32mScaling Generative AI Inference with llm-d
33mBeyond VLLM: Distributed LLM Inferencing With Llm-d on Kubernetes
08mKeynote: Plug in and Scale: Serving LLM Models on Kubernetes Ma...
38mOptimizing LLM Inference for the Rest of Us
31mThe Evolution of Multi-GPU Inference in vLLM
36mCloud-Native Model Serving: vLLM's Lifecycle in Kubernetes
31mFrom Model Serving To Distributed Inference: How Llm-d Evolves AI Platforms on…