Fast Inference, Furious Scaling: Leveraging VLLM With KServe
summary
vLLM claims to cut KV cache memory waste from 60–80% to about 4% by using vLLM paged attention, plus continuous batching and custom CUDA kernels to boost throughput. KServe provides Kubernetes-native autoscaling, a local model cache to warm models and LMCache to offload KV state. Demo shows Kind + KServe + vLLM running a 1B-parameter Llama 3.2 locally, loading and serving requests (prompt 8 tokens, completion 5 tokens), so the integration works end to end.
AI-written summary · about
chapters (6)
- 0:00Introduction
- 1:05LLM serving challenges
- 8:14Introducing VLLM
- 12:31KServe scaling
- 15:02Local demo setup
- 22:51Closing
related talks
33mBeyond VLLM: Distributed LLM Inferencing With Llm-d on Kubernetes
24mKServe Next: Advancing Generative AI Model Serving
05mEvolving KServe: The Unified Model Inference Platform For Both...
26mLMCache: Lower LLM Performance Costs in the Enterprise
20mInferencing LLMs in production with Kubernetes and KubeFlow
29mServing the Future: KServe’s Next Chapter Hosting LLMs & GenAI Models (with Fun Drawings!)
32mScaling Generative AI Inference with llm-d
36mCloud-Native Model Serving: vLLM's Lifecycle in Kubernetes