MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs
summary
Production system trains LLMs across >10,000 GPUs using a distributed profiler plus device-level torch profiling, fine-grained intra-node trunking (8 trunks) and communication-overlap for 3D parallelism, yielding about 1.3x speedup over Megatron and claiming a 300B-token run in under two days. A robust training framework automates checkpointing, GPU health checks and eviction, isolates stragglers, finds issues like CPU memory leaks, and handles ~90% faults automatically while ~10% of failures still need human analysis.
AI-written summary · about
chapters (13)
- 0:00MegaScale introduction
- 1:36Training high-quality models
- 4:28Training efficiency challenges
- 6:40Infrastructure profiling
- 8:31In-node communication with Megatron
- 10:22Inter-node communication overlap
- 11:46Performance results
- 12:50Stability issues in clusters
- 13:55Fault-tolerant training framework
- 16:28Cluster case studies
- 18:18Future model architectures
- 19:03Elasticity and data corruption
- 20:27Summary
related talks
16mMegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs
24mEfficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM
23mOperationalizing Large Language Model Training Pipelines
19mSystem for Scaling Large Language Models
28mTraining Large Language Models on Kubernetes
22mPerformance Optimizations at 100K+ Scale
17mCharacterization of Large Language Model Development in the Datacenter
15mResource Multiplexing in Tuning and Serving Large Language Models