Serving the Future: KServe’s Next Chapter Hosting LLMs & GenAI Models (with Fun Drawings!)
Talkat Cloud Native + Kubernetes AI Day Europe 2025 · on Tuesday, 2025-04-01· at 10:40· Level 1 | Hall Entrance N10 | Room G·
summary
KServe now ships concrete features for GenAI: a local model cache CRD that auto-downloads models to node PVs and self-heals, OpenAI protocol endpoints for completions/chat/embeddings, and LM metric-based autoscaling that uses token throughput or time‑to‑first‑token with real‑time metrics. Multi-node inference uses Ray + VLM for models beyond single‑node memory, prompt/KV caching and an LM cache reduce time‑to‑first‑token by ~3–10×, and an Envoy AI gateway enables hybrid routing, token rate‑limiting and unified model routing.
AI-written summary · about
chapters (15)
- 0:00KServe and GenAI challenges
- 3:11What is KServe
- 6:40New GenAI features
- 8:27KServe GenAI architecture
- 8:49Model caching for scaling
- 11:58Autoscaling LLMs by metrics
- 13:49OpenAI protocol in KServe
- 14:10Multi-node VLM inference
- 15:15Prompt caching benefits
- 17:02Envoy AI Gateway
- 19:28Gateway API for GenAI
- 21:14Batch inference optimization
- 21:36Prefill vs decoding stages
- 22:40Enterprise scale lessons
- 24:46Q&A
related talks
34mKServe in Production: Scaling Generative Inference the Cloud Native Way
05mEvolving KServe: The Unified Model Inference Platform For Both...
23mFast Inference, Furious Scaling: Leveraging VLLM With KServe
28mExploring ML Model Serving with KServe (with fun drawings)
40mServing Machine Learning Models at Scale Using KServing
31mFrom Model Serving To Distributed Inference: How Llm-d Evolves AI Platforms on…
31mServing Machine Learning Models at Scale Using KServe
36mWhat's New, ModelMesh? Model Serving at Scale