AI Model Inference on K8s
summary
Explains how Kubernetes plus KServe/model mesh runs real inference: single-model KNative serverless pods and multi-model deployment that packs thousands of models into few servers, with a concrete claim of 20,000 models deployed on two pods (small string models). Covers runtimes (TensorFlow, scikit‑learn, OpenVINO), built‑in pre/post processors like transformer and explainer and ensembler, and work on Open Inference Protocol v2 and attachable model containers to handle 500+ GB LLMs for faster cold start.
AI-written summary · about
chapters (11)
- 0:00AI model serving importance
- 1:05From model writing to serving
- 2:08KServe and model servers
- 3:51Model serving like Java apps
- 4:55KServe inference platform
- 6:18Single vs multi-model deployment
- 7:23Multi-model resource savings
- 9:11KServe advanced features
- 10:34LLM challenges on Kubernetes
- 12:40Contributing to KServe
- 13:23KServe demo and setup
related talks
34mKServe in Production: Scaling Generative Inference the Cloud Native Way
10mBuilding Machine Learning Inference Through Knative Serverless...
35mAdvanced Model Inferencing Leveraging KNative, Istio & Kubeflow Serving
28mExploring ML Model Serving with KServe (with fun drawings)
35mServing Large Language Models using KServe
23mEffortless Scalability: Orchestrating Large Language Model Inference with Kubernetes
40mServing Machine Learning Models at Scale Using KServing
31mFrom Model Serving To Distributed Inference: How Llm-d Evolves AI Platforms on…