Problems and Solutions with AI Model Deployments in k8s
summary
Compares three Kubernetes deployment patterns for AI models: baking models into container images, storing models in Git LFS, and downloading models at startup from Google Cloud Storage (GCS), plus using image streaming via GKE/Artifact Registry. Measures build and cold-start times (baked images ~5m42s builds and slow pulls; Git LFS hits 2–5 GB file limits and billing issues; GCS downloads ~1.5 min cold starts with faster builds), and reports that enabling image streaming makes cold starts near-instant. Describes using Docker cache, Triton for inference, GPU nodes, and the practical operational limits discovered.
AI-written summary · about
chapters (17)
- 0:00AI model deployment intro
- 1:07AI dev problems in k8s
- 2:35Training vs deployment
- 5:58Deployment approaches
- 7:32Download models from bucket
- 9:06Bucket download drawbacks
- 12:01Models in Docker image
- 13:52Git LFS storage issues
- 16:07Optimize Git LFS costs
- 18:11Models in GCS bucket
- 20:27Bucket storage speed issues
- 21:31Google Cloud Image Streaming
- 23:44Demo: three approaches
- 29:37Image build speed test
- 30:46App startup time test
- 33:52Conclusion
- 35:00Q&A
related talks
38mAI in Kubernetes: How to Get Started?
21mOptimizing Model Serving on Kubernetes With Model Streaming
14mAI Deployment: Mastering LLMs with KFServing in Kubernetes
17mChallenges and takeaways of managing AI workloads on cloud
27mHarnessing Kubernetes for Scalable AI/ML Workloads
33mAI Model Distribution Challenges and Best Practices
31mThe Future of Data on Kubernetes: From Database Management To AI Foundation
25mSecuring AI Pipelines: Real-World Attacks on Kubernetes-Based AI Infrastructure