Shared GPU Scheduling & Proactive Autoscaling: A Production Blueprint for 1000+…
summary
Production platform integrates Hami vGPU virtualization with a KEDA extension exposing a KEDA Consumer Saturation metric (unacked messages / active consumers) so autoscaling triggers at 1.7 saturation to absorb a ~60s model warm‑up. Implementation uses a scheduler extender, Prometheus metrics, and Helm GitOps multi-region deployment across 1,000+ A100 GPUs and 1,200 workflows. Result: ~55% less GPU time, 91% faster recovery, 85% fewer GPU errors during peaks, and seamless 7× cloud burst with no user-visible downtime.
AI-written summary · about
chapters (10)
- 0:00Shared GPU scheduling intro
- 1:24GPUs in Kubernetes challenges
- 8:06Snow Corp production deployment
- 11:57Architecture and CNCF integration
- 13:22GPU sharing and live migration
- 15:52Proactive autoscaling for volatile workloads
- 18:43Results and business impact
- 20:50Key takeaways and CNCF case study
- 22:13Adoption and future roadmap
- 23:44Q&A
related talks
24mCluster Management for Large Scale AI and GPUs: Challenges and Opportunities
22mKeep HPC Running - an SRE's Guide to Supporting GPUs on Kubernetes
28mGPU Reservations: Maximizing Utilization and Fairness Across Teams
41mBeyond Uptime: Revolutionizing Fintech Reliability
43mThe GPU Utilization Problem: What’s Going Wrong and How to Solve It
21mExperience Operating Large GPU Clusters at Organizational Scale
10mLightning Talk: K8s Issue #52757: Sharing GPUs Among Multiple Containers
05mHami: Dynamic, Smart, Stable GPU-Sharing Middleware In Kubern...