Consume and Serve Using Caikit and ModelMesh
summary
Caikit provides a runtime-and-module toolkit with ready data models and a GitHub template that wraps Hugging Face pipelines into simple REST/gRPC endpoints and a live Gradio demo that actually works; ModelMesh is a KServe multimodel backend that loads and unloads models to and from memory, treats pods as a distributed LRU cache and removes the Kubernetes pod-per-model limit, having been used in production for roughly 8–9 years under Watson services. The demo wires Caikit to ModelMesh, shows object detection and sentiment examples, and confirms runtime placement, autoscaling and retries function end to end.
AI-written summary · about
chapters (17)
- 0:00Introduction
- 0:22Model serving importance
- 1:06Deployment challenges
- 2:53Introducing KServe
- 4:40Scaling instances
- 5:22Model Mesh overview
- 7:30Model caching and placement
- 8:36Consuming models easily
- 10:43Introducing Caikit
- 11:49AI development roles
- 14:18Caikit modules and runtimes
- 17:15Demo setup and GitHub
- 20:43Live demo with Hugging Face
- 24:13Creating Caikit project
- 25:58Testing Caikit APIs
- 27:45Q&A
- 28:28Future directions
related talks
36mWhat's New, ModelMesh? Model Serving at Scale
40mServing Machine Learning Models at Scale Using KServing
29mIntegrating High Performance Feature Stores with KServe Model Serving
28mExploring ML Model Serving with KServe (with fun drawings)
32mEnhancing the Performance Testing Process for gRPC Model Inferencing at Scale
31mServing Machine Learning Models at Scale Using KServe
18mBaby Sitting Your AI Models in Production? STOP!
39mIntroducing KFServing: Serverless Model Serving on Kubernetes