Characterization of Large Language Model Development in the Datacenter
Talkat 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI '24) · 2024-04-16..04-18·
summary
Six-month datacenter trace from Acme covers >4,000 A100 GPUs and ~1M jobs (7B–123B model family, 60% GPU jobs) and shows most GPU time coming from large jobs (>256 GPUs use ~96% of GPU-time) and polarized utilization. The team profiles pretraining with Inter Evo and reports ~16% throughput gain from v2, builds fault-tolerant pretraining with host-memory synchronous checkpoints (up to 58× faster checkpointing) plus automated failure diagnosis, finds infrastructure faults cause most waste, and implements decoupled scheduling for evaluation to cut idle and raise throughput 1.3–1.8×.
AI-written summary · about
chapters (8)
- 0:00Introduction and motivation
- 2:13Data center infrastructure
- 3:42Workload characterization
- 7:29Workload profiling
- 10:21Evaluation efficiency
- 11:37Failure analysis
- 14:03System-level optimizations
- 16:09Conclusion
related talks
19mAnalysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads
15mMLaaS in the Wild: Workload Analysis and Scheduling in Large-Scale Heterogeneous GPU...
21mMegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs
23mOperationalizing Large Language Model Training Pipelines
24mEfficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM
22mGenAI Training In Production: Software, Hardware & Network Considerations
28mComputing Large Foundational Models
16mUnderstanding Stragglers in Large Model Training Using What-if Analysis