From Model Serving To Distributed Inference: How Llm-d Evolves AI Platforms on…
summary
Platform teams get a concrete reference architecture: KAITO performs GPU node auto-provisioning, model onboarding, OpenAI‑compatible endpoints and autoscaling while llm-d adds inference‑aware scheduling, KV‑cache‑aware routing, cross‑node cache coordination, and prefill/decode disaggregation. Benchmarks claim ~90% KV‑cache hit rates and P/D disaggregation roughly doubles tokens per GPU. A production stack integrates the Gateway API Inference Extension and Kaito, runs 70+ end‑to‑end tests with real and mocked GPUs, and demonstrates workable distributed inference on Kubernetes.
AI-written summary · about
chapters (17)
- 0:00Distributed LLM inference intro
- 1:02Distributed inference challenges
- 4:43Production inference metrics
- 5:47LLM inference reference arch
- 7:33AMD for distributed inference
- 9:21AMD features and performance
- 10:03KV cache management in AMD
- 13:55Hierarchical KV cache tiers
- 15:42Model of Experts and EP support
- 17:06Profile-decode separation
- 19:34Kaito workload lifecycle
- 21:41Kaito GPU provisioning
- 23:46Kaito capacity planning
- 25:15Kaito auto-scaling with KEDA
- 27:47Kaito and LLM-D integration
- 29:12GPU mocking for inference tests
- 30:34Q&A
related talks
34mKServe in Production: Scaling Generative Inference the Cloud Native Way
08mKeynote: Plug in and Scale: Serving LLM Models on Kubernetes Ma...
33mllm-d: Kubernetes Native Distributed Inferencing
24mKServe Next: Advancing Generative AI Model Serving
32mScaling Generative AI Inference with llm-d
29mServing the Future: KServe’s Next Chapter Hosting LLMs & GenAI Models (with Fun Drawings!)
27mServing PyTorch LLMs at Scale: Disaggregated Inference With Kubernetes and Llm-d
23mEffortless Scalability: Orchestrating Large Language Model Inference with Kubernetes