Fast Inference, Furious Scaling: Leveraging VLLM With KServe
summary
vLLM claims major inference gains by cutting KV‑cache waste from roughly 60–80% down to ~4% using PagedAttention, plus speedups from continuous batching and custom CUDA kernels. KServe provides Kubernetes autoscaling, a local model cache to warm replicas, and KV‑cache offloading (LM Cache). Demo stitches vLLM into KServe on a local kind cluster, pulls a Llama 3 3.21B HF model, runs on CPU with ~0.2 tokens/sec on a laptop but successfully serves requests end-to-end.
AI-written summary · about
chapters (6)
- 0:00Introduction
- 1:25LLM serving challenges
- 8:42VLLM for faster inference
- 13:19KServe scaling and management
- 16:17Demo: VLLM + KServe
- 27:40Resources and closing
related talks
34mImprove AI Inference (serving models) With KServe and VLLM
24mKServe Next: Advancing Generative AI Model Serving
33mBeyond VLLM: Distributed LLM Inferencing With Llm-d on Kubernetes
34mKServe in Production: Scaling Generative Inference the Cloud Native Way
36mCloud-Native Model Serving: vLLM's Lifecycle in Kubernetes
20mInferencing LLMs in production with Kubernetes and KubeFlow
33mllm-d: Kubernetes Native Distributed Inferencing
26mLMCache: Lower LLM Performance Costs in the Enterprise