Enhancing the Performance Testing Process for gRPC Model Inferencing at Scale
Talkat Kubernetes AI Day Europe 2022 · on Monday, 2022-05-16· at 14:10· Pavilion 3, Room D | Level 2 | Central Forum·
summary
Shows a Kubernetes setup using KServe + ModelMesh to host high-density gRPC inference, with ModelMesh paging models in and out of memory as an LRU cache and coordinating Triton runtimes. Performance automation uses a k6 load driver sending gRPC traffic, metrics stored to Influx/Prometheus and visualized in Grafana. Demonstrates running 10k registered models, achieving about 2k requests/sec for small models with single-digit to ~20 ms latencies, and reproduces cache-miss behavior when loading cold models.
AI-written summary · about
chapters (21)
- 0:00Introduction
- 0:53Model Deployment
- 1:41Kubernetes
- 2:30Complexities
- 3:04Kserve
- 4:59Scalability
- 6:12Model Mesh
- 8:58Model Mesh Features
- 10:28Performance Testing Automation
- 11:54Performance Testing Setup
- 13:16Performance Testing Environment
- 14:13QFlow Pipeline
- 14:38K6 Load Tools
- 15:37GRPC
- 16:09Prometheus
- 16:45Demo
- 19:01Testing
- 20:18Testing Log
- 21:12Testing Results
- 25:27Cashmiss Action
- 27:11Should I use Model Mesh
related talks
40mServing Machine Learning Models at Scale Using KServing
29mIntegrating High Performance Feature Stores with KServe Model Serving
36mWhat's New, ModelMesh? Model Serving at Scale
31mServing Machine Learning Models at Scale Using KServe
34mKServe in Production: Scaling Generative Inference the Cloud Native Way
35mServing Large Language Models using KServe
15mAI Model Inference on K8s
23mEffortless Scalability: Orchestrating Large Language Model Inference with Kubernetes