Exploring ML Model Serving with KServe (with fun drawings)
Talkat Kubernetes AI Day Europe 2022 · on Monday, 2022-05-16· at 10:55· Pavilion 3, Room D | Level 2 | Central Forum·
summary
KServe is a Kubernetes CRD that actually deploys and monitors inference services using Knative/Envoy (Istio) for routing, supports scale-to-zero with GPU/CPU autoscaling and request batching, and exposes a standard inference protocol (HTTP/gRPC) so runtimes (TensorFlow, PyTorch, Triton, SKLearn, XGBoost) are swappable. It adds an inference graph (sequence/switch/splitter/ensemble), explainers, async logging and detectors (drift/outlier/adversarial), and a multi-model ModelMesh puller sidecar that pulls models on demand; demoed canary rollouts (10% → promote) working end-to-end.
AI-written summary · about
chapters (17)
- 0:00Introduction
- 7:40Features
- 8:20Predictor
- 9:40Control Plane
- 10:24Replicas
- 11:09CoopTree
- 11:46Data Plane
- 12:20Inference Graph
- 13:18Standard Inference Protocol
- 14:01Data Plane Plugins
- 15:19Monitor Logger
- 16:53Serving Runtime
- 17:35MultiModel Serving
- 19:43Use Cases
- 21:34Inference Service
- 23:34Conclusion
- 24:18Questions
related talks
40mServing Machine Learning Models at Scale Using KServing
24mKServe Next: Advancing Generative AI Model Serving
35mServing Large Language Models using KServe
34mKServe in Production: Scaling Generative Inference the Cloud Native Way
31mServing Machine Learning Models at Scale Using KServe
39mIntroducing KFServing: Serverless Model Serving on Kubernetes
15mAI Model Inference on K8s
29mServing the Future: KServe’s Next Chapter Hosting LLMs & GenAI Models (with Fun Drawings!)