Inferencing LLMs in production with Kubernetes and KubeFlow
summary
A production pipeline uses Kubernetes on GKE with Kubeflow pipelines, KServe as the serverless model layer, KEDA for event-driven scaling and a VLM layer to treat GPU memory as virtual pages. The build serves multitenant customer‑support LLMs on-prem or cloud, adds canary traffic splits, batch jobs and fine‑tuning pipelines, and claims to eliminate the typical 60–80% KV cache GPU memory overhead so throughput and GPU utilization rise markedly. KServe + VLM is the concrete stack.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 0:49Real World Problem
- 4:49Case Study
- 6:33Foundation
- 8:22Queserve
- 9:55Use cases
- 10:49VM layer
- 12:50Pipelines
- 15:02Key takeaways
- 16:01QA Session
related talks
23mFast Inference, Furious Scaling: Leveraging VLLM With KServe
26mStreamlining AI Pipelines With Elyra: From Development To Inference With KServe & VLLM
33mFast Inference, Furious Scaling: Leveraging VLLM With KServe
33mBeyond VLLM: Distributed LLM Inferencing With Llm-d on Kubernetes
08mKeynote: Plug in and Scale: Serving LLM Models on Kubernetes Ma...
33mllm-d: Kubernetes Native Distributed Inferencing
34mImprove AI Inference (serving models) With KServe and VLLM
34mKServe in Production: Scaling Generative Inference the Cloud Native Way