Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads
Talkat USENIX ATC '19 · on Friday, 2019-07-12· at 11:25· USENIX ATC Track II: Grand Ballroom VII–IX·
summary
Analysis uses a 75‑day trace from Microsoft Philly (shared by 14 production groups) combining scheduler logs, hardware counters and AI‑engine logs to profile ~996K training jobs. Key findings: median single‑GPU utilization is ≈65%, utilization falls ~30% as jobs span servers due to distribution/locality loss, and failed jobs consume ≈73% of GPU hours lost (many jobs see >1 failure). A rule‑based failure classifier parses stdout/stderr to attribute failures and the paper advocates locality‑aware scheduling, early validation and migration to cut wasted GPU time.
AI-written summary · about
chapters (16)
- 0:00Intro
- 0:32Deep Learning at a Large Enterprise
- 1:52Job Lifecycle in Philly
- 2:29Contributions
- 3:21Study Details
- 4:37GPU Utilization for Job Sizes
- 5:43Effect of Distribution on Dedicate Servers
- 6:31Scheduling Distributed Training
- 7:47Failures Can Reduce Cluster Utilization
- 8:18Challenge: Failures across Stack
- 8:48Failure Classifier
- 9:58Failures in High Frequency
- 10:37Failures in High Resource Use
- 11:28Locality v.s. Waiting Time
- 13:00Job Pre-Run before Scheduling
- 14:19Conclusion
related talks
15mMLaaS in the Wild: Workload Analysis and Scheduling in Large-Scale Heterogeneous GPU...
43mThe GPU Utilization Problem: What’s Going Wrong and How to Solve It
17mCharacterization of Large Language Model Development in the Datacenter
24mCluster Management for Large Scale AI and GPUs: Challenges and Opportunities
16mTowards GPU Utilization Prediction for Cloud Deep Learning
01h 15mTutorial: Build, Operate, and Use a Multi-Tenant AI Cluster Based Entirely on Open Source
20mHow are You Utilizing GPUs? Best Practices on Managing GPUs in Azure
41mPanel: AI Infra Best Practices: Enterprise Do’s and Don’ts