Beyond VLLM: Distributed LLM Inferencing With Llm-d on Kubernetes
summary
Deploys vLLM pods on Kubernetes/OpenShift with KServe, Envoy and GuideLLM for benchmarking, then inserts llm-d as an intelligent router with an endpoint picker (EPP) that scrapes pods, filters unhealthy ones and scores for KV-cache-aware routing. Demo shows KV cache hit rate rising from under 50% to >90%, P95 time-to-first-token improving ~38% at 32 clients and ~52% at 64, and first-token latency nearly doubling improvement (≈98%), demonstrating multi-node multi‑GPU distribution works in practice.
AI-written summary · about
chapters (13)
- 0:00LLMD and vLLM for LLM inference
- 1:05Single model serving with vLLM
- 8:25vLLM’s key innovations
- 10:34Live demo: vLLM on OpenShift AI
- 14:43Benchmarking vLLM with Guide LLM
- 16:06vLLM prefill and decode phases
- 20:00KV cache hit rate importance
- 21:49Scaling LLM inference challenges
- 22:50LLMD integrates Kubernetes and vLLM
- 25:59LLMD endpoint picker routing
- 29:33LLMD vs load balancer performance
- 31:52LLMD advanced features and future
- 32:56Summary
related talks
33mllm-d: Kubernetes Native Distributed Inferencing
23mFast Inference, Furious Scaling: Leveraging VLLM With KServe
32mScaling Generative AI Inference with llm-d
40mScalable and Efficient LLM Serving With the VLLM Production Stack
33mFast Inference, Furious Scaling: Leveraging VLLM With KServe
34mImprove AI Inference (serving models) With KServe and VLLM
42mStop Treating LLMs Like REST APIs
27mServing PyTorch LLMs at Scale: Disaggregated Inference With Kubernetes and Llm-d