MLaaS in the Wild: Workload Analysis and Scheduling in Large-Scale Heterogeneous GPU...
Talkat 19th USENIX Symposium on Networked Systems Design and Implementation (NSDI '22) · 2022-04-04..04-06·
summary
A two-month Alibaba MLaaS trace covers over 6,700 GPUs across ~1,800 nodes with 1.2M tasks, 7.5M instances and shows concrete pain points: 85% gang-scheduled jobs, widespread over-requesting of CPU/GPU, low GPU utilization, and long queue delays for high-end GPUs. Measured gains include GPU sharing saving ~2,500 GPUs at peak, duration prediction that cuts average completion time by about 76–77%, and reserving/packing that reduces v100 queuing delay by ~68%; CPU contention often explains slowdowns.
AI-written summary · about
chapters (11)
- 0:00Intro
- 0:37Production ML Workloads
- 1:53Trace Overview
- 3:03Run-time and Queueing delays
- 3:42Resource Requests & Usage
- 4:46Machine Resource Utilization
- 5:56GPU Sharing
- 7:32Duration Predict for Recurring Tasks
- 9:17CPU can be the bottleneck
- 11:18Imbalanced Scheduling
- 13:56Takeaways
related talks
19mAnalysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads
17mCharacterization of Large Language Model Development in the Datacenter
20mScheduler and Sharding Considerations for Network Efficiency
17mServing Heterogeneous Machine Learning Models on Multi-GPU Servers...
24mCluster Management for Large Scale AI and GPUs: Challenges and Opportunities
43mThe GPU Utilization Problem: What’s Going Wrong and How to Solve It
21mExperience Operating Large GPU Clusters at Organizational Scale
44mScaling ML from 0 to millions of users