GREYHOUND: Hunting Fail-Slows in Hybrid-Parallel Training at Scale
summary
Characterization on a 10,000+ GPU multi-tenant cluster finds transient fail-slows from CPU contention, GPU thermal throttling, and network congestion that last sub-minutes to hours, delay large jobs by ~1.34× on average and affect ~60% of >512-GPU jobs (communication stragglers ~40% of probes, avg 24 minutes; large-job avg 72 minutes). GREYHOUND uses LD_PRELOAD-based non-intrusive hooks to track iteration time, a three-phase tracking/profiling/validation pipeline, an online ski‑rental mitigation (S1–S4), 99%+ detection accuracy with 0.39% overhead and ~10s reaction, yielding 1.58× end-to-end throughput.
AI-written summary · about
chapters (15)
- 0:00Greyhound introduction
- 1:00Hybrid parallel training
- 3:30Struggler characteristics
- 5:57Computation strugglers causes
- 7:25Communication strugglers causes
- 8:30Impact of strugglers
- 9:12Detection system requirements
- 10:35Greyhound detection design
- 12:21Detecting strugglers with BCD
- 12:42Mitigation requirements
- 13:04Mitigation design space
- 14:27Adaptive mitigation
- 15:08Greyhound evaluation
- 16:54Greyhound at 256 GPUs
- 17:38Summary
related talks
19mAnalysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads
16mUnderstanding Stragglers in Large Model Training Using What-if Analysis
17mServing Heterogeneous Machine Learning Models on Multi-GPU Servers...
45mHigh-Performance and Scalable Middleware for HPC and Deep...
15mMLaaS in the Wild: Workload Analysis and Scheduling in Large-Scale Heterogeneous GPU...
17mCharacterization of Large Language Model Development in the Datacenter
43mThe GPU Utilization Problem: What’s Going Wrong and How to Solve It
26mDeepSpeed: Efficient Training Scalability for Deep Learning