From Minutes to Seconds: How we reduced ML models cold start time by 90% on Kubernetes
summary
Claim: cold-start time drops ~90% from ~11 minutes to ≈1 minute by combining Kubernetes + KServe with node prep and image/model preloading. Changes: increase EBS throughput (125→500 Mbps cuts image pull by ~3 minutes), use warm pools to boot and stop pre-initialized GPU nodes, preload container images and OCI image volumes for instant model mounts, and tweak GPU operator validation. Built small cops integrations, fixed two bugs, contributed patches upstream; result is consistent sub‑minute starts without new infra and modest extra EBS cost.
AI-written summary · about
chapters (17)
- 0:00Introduction and overview
- 1:56Kubernetes setup for ML inference
- 4:22Cold start time challenges
- 6:34Investigating node bootstrapping delays
- 9:02Optimizing image pulling speed
- 11:59Preloading images in AMIs
- 14:08Reserved GPU capacity limitations
- 15:34Warm pools concept and benefits
- 18:27Implementing warm pools with Cops
- 21:58Preloading images during warm phase
- 24:09GPU driver installation challenges
- 26:55GPU operator validation improvements
- 27:18Model pulling bottlenecks
- 28:19Mounting models as OCI volumes
- 30:08Final results and 90% reduction
- 31:17Contributing changes upstream
- 32:19Q&A session
related talks
11mLightning Talk: Roche's Kubeflow Story: Large Models in Seconds, Costs Cut, Securit...
34mWhere's My Pod, Already? A Tale of Cold-Start Struggles on Kubernetes
43mProblems and Solutions with AI Model Deployments in k8s
25mFrom Cold Start to Warp Speed: Triton Kernel Caching with OCI Container images
07mLightning Talk: Slash LLM Cold-Start Times by Pre-distributing GPU...
37mFrom Hours To Milliseconds: Scaling AI Inference 10x With Serverless on Kubernetes
11mFast and Efficient Container Startup at the Edge via Dependency Scheduling
31mGotta Cache 'em All: Scaling AI Workloads With Model Caching in a Hybrid...