What's New, ModelMesh? Model Serving at Scale
summary
KServe’s ModelMesh serves as a Kubernetes multi-model backend that hosts 60,000 models managed while keeping only about 5,000 in memory, using a distributed LRU cache to load and evict models and retry failed loads. It uses etcd for metadata, exposes Prometheus metrics with a Grafana dashboard (shows cache miss latency up to 1.5s cold-load), and supports runtimes like TorchServe and MLServer plus custom Python runtimes via MLServer; production use includes IBM Watson services.
AI-written summary · about
chapters (20)
- 0:00Introduction
- 1:59Outline
- 2:55What is Model Serving
- 3:48Deploying Models as Microservices
- 5:33Project Kserv
- 7:07Pod Per Model Paradigm
- 9:06Model Mesh
- 11:30Model Mesh Architecture
- 13:23Model Mesh Architecture Overview
- 14:57Monitoring Model Mesh
- 16:00Prometheus
- 16:58ModelMesh Dashboard
- 18:22Cache Miss Rate
- 19:45Serving Runtimes
- 21:56Example
- 22:53Custom Runtime
- 24:04Inference Service
- 25:27Demo
- 29:26Contact Information
- 30:48Questions
related talks
31mServing Machine Learning Models at Scale Using KServe
40mServing Machine Learning Models at Scale Using KServing
28mExploring ML Model Serving with KServe (with fun drawings)
31mConsume and Serve Using Caikit and ModelMesh
29mIntegrating High Performance Feature Stores with KServe Model Serving
35mServing Large Language Models using KServe
32mEnhancing the Performance Testing Process for gRPC Model Inferencing at Scale
27mModel Serving at the Edge Made Easier