Improve AI Inference (serving models) With KServe and VLLM
summary
KServe and vLLM combine to serve LLMs on Kubernetes with concrete wins: KServe provides serverless scale-to-zero and multi-model packing, vLLM adds compression and quantization (via LLM Compressor) that cuts memory use and raises throughput — presenter cites a 510-model catalog and a wider 63% industry uptake of open-source models — techniques shown include continuous batching, pagedAttention and speculative decoding and a new LLMD router for large-scale model pools.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 3:40Why serving AI models matters
- 7:39Open source AI models matter
- 12:18KServe for AI on Kubernetes
- 15:35VLLM for scalable LLMs
- 19:09VLLM features and quantization
- 23:30LLMD for LLM management
- 25:19Red Hat’s AI platform
- 26:44Q&A
related talks
33mFast Inference, Furious Scaling: Leveraging VLLM With KServe
24mKServe Next: Advancing Generative AI Model Serving
33mBeyond VLLM: Distributed LLM Inferencing With Llm-d on Kubernetes
08mKeynote: Plug in and Scale: Serving LLM Models on Kubernetes Ma...
36mCloud-Native Model Serving: vLLM's Lifecycle in Kubernetes
35mServing Large Language Models using KServe
34mKServe in Production: Scaling Generative Inference the Cloud Native Way
20mInferencing LLMs in production with Kubernetes and KubeFlow