Advancements in AI/ML Inference Workloads on Kubernetes From WG Serving and Ecosystem Projects
Talkat KubeCon + CloudNativeCon Europe 2025 · on Wednesday, 2025-04-02· at 11:15· Level 3 | ICC Capital Suite 7-9·
summary
WG Serving reports one year of work: it ships an inference-perf benchmark library (Python, VRM support, CR GPT dataset) to identify autoscaling metrics and token/prompt effects; prototypes image-volume-as-data to avoid pulling 100s‑GB model images; advances multi‑host patterns via leader‑worker integrations and Kerf CRD proposals; pushes Kubernetes DRA and device‑failure APIs before GA; and releases gateway API inference extension 0.1 to reduce tail latency and improve throughput on shared foundation models. Community meetings and repos are active.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 2:29Working group goals
- 7:09Autoscaling and benchmarking
- 8:37Multi-host workloads
- 10:00Dynamic resource allocation
- 12:10Gateway API extension
- 13:16Benchmarking tool overview
- 15:44Serving catalog patterns
- 18:58Community engagement
- 21:49How to contribute
- 23:40Q&A
related talks
34mKServe in Production: Scaling Generative Inference the Cloud Native Way
23mEffortless Scalability: Orchestrating Large Language Model Inference with Kubernetes
24mKServe Next: Advancing Generative AI Model Serving
20mOperating Frontier Intelligence at Scale
38mKubernetes for Multi-Host Training and Inference: Workload Aware Sc...
15mAI Model Inference on K8s
31mFrom Model Serving To Distributed Inference: How Llm-d Evolves AI Platforms on…
08mKeynote: Plug in and Scale: Serving LLM Models on Kubernetes Ma...