You Got a Match! LLM Prefix Aware Routing With Kubernetes
summary
Kubernetes extension implements an inference gateway that routes LLM requests by initial token prefixes and maintains a global cache index to avoid GPU re‑computation. Built as a Gateway API Inference Extension plugin and used with LLMD for PD (prefill/decode) split, the router reports up to 90% cache hits, 500x faster time‑to‑first‑token and 3x throughput versus round‑robin in benchmarks and supports hash‑based prefix indexing, scorer plugins, and dynamic load+affinity balancing.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 0:44Gateway API extension
- 3:55Kubernetes native inference
- 6:26Cache misses in inference
- 8:17Prefix aware routing benefits
- 9:24Smart inference routing evolution
- 11:57Routing strategies for cache
- 14:49Estimated cache index
- 17:37Cache affinity vs load
- 19:21Weighted scoring and ML routing
- 20:26Benchmarking results
- 22:54Key takeaways
- 23:37Q&A
related talks
33mllm-d: Kubernetes Native Distributed Inferencing
33mBeyond VLLM: Distributed LLM Inferencing With Llm-d on Kubernetes
16m⚡ Lightning Talk: Introducing LLM Instance Gateways for Efficient Inference Serving
38mOptimizing LLM Inference for the Rest of Us
42mStop Treating LLMs Like REST APIs
14mSponsored Session: Lightning Talk: Efficient Inference Serving with Kubernetes Gateway...
32mScaling Generative AI Inference with llm-d
30mSecuring LLMs in Kubernetes: Best Practices