Real-Time GPU Job Scheduling Latency Prediction in Multi-Cluster Kubernetes
summary
Bloomberg runs a Karmada controller that computes live GPU queue ETAs by tracking scheduler objects and measuring the delta from creation to last-transition timestamps, maintaining an O(1) rolling average per size tier (small/medium/large, large >10 GPUs) using a rolling window of 100 samples with a hydration step on restart. Outputs are published to Grafana/Prometheus and returned at submission via a webhook; published metrics are raw latency, rolling average, pending count and the system reduces resubmits and over‑requests while keeping computation constant time.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 1:24User pain points
- 2:27GPU job scheduling
- 3:50Why average latency failed
- 5:36Bucketing by GPU size
- 8:05Clean scheduling signals
- 9:29Real-time rolling average
- 10:54User experience before/after
- 11:56Handling controller restarts
- 13:01Publishing metrics
- 14:46Latency tracking flow
- 16:49Key lessons learned
- 18:34Future plans
- 20:18Conclusion and Q&A
related talks
28mAI Workload Preemption in a Multi-Cluster Scheduling System at Bloomberg - Leon Zhou & Wei-Cheng Lai
33mFrom spreadsheet scheduling to Kubernetes: building an on-premise ML platform
27mEfficient Multi-Cluster GPU Workload Management with Karmada and Volcano
24mCluster Management for Large Scale AI and GPUs: Challenges and Opportunities
31mShared GPU Scheduling & Proactive Autoscaling: A Production Blueprint for 1000+…
10mFair Scheduling for Deep Learning Workloads in Kubernetes
36mCapacity Scheduling for Elastic Resource Sharing in Kubernetes
43mThe GPU Utilization Problem: What’s Going Wrong and How to Solve It