AI Deployment: Mastering LLMs with KFServing in Kubernetes
summary
Shows a concrete KFServing pipeline that stages models from object storage via a storage initializer, serves LLMs on Kubernetes with optional GPU nodes, and runs automated canary traffic shifts between Model A and Model B driven by latency and accuracy metrics. Includes how to collect those metrics, roll back or promote models automatically, and the extra CPU/GPU cost and operational complexity experienced in practice, with examples of what failed and what stabilized performance.
AI-written summary · about
chapters (10)
- 0:00AI deployment challenges
- 1:03Kubernetes and KFServing
- 2:56Model serving experiments
- 4:24KFServing architecture
- 6:56Canary and A/B testing
- 8:30Traffic and metrics
- 10:17Multiple models in Kubernetes
- 11:23GPU optimization for inference
- 12:31Security for model traffic
- 13:15Q&A
related talks
08mKeynote: Plug in and Scale: Serving LLM Models on Kubernetes Ma...
35mServing Large Language Models using KServe
39mIntroducing KFServing: Serverless Model Serving on Kubernetes
25mScaling Machine Learning Models on Kubernetes
43mProblems and Solutions with AI Model Deployments in k8s
32mRunning and Fine-tuning Open Source LLMs on Google Kubernetes Engine
46mDeploying scalable Machine Learning models with Kubernetes
30mAccelerate Federated Learning Model Deployment with KServe (KFServing)