Lightning Talk: Trinity Large - Torchtitan on 2000+ B300s
summary
Describes training a 400B MoE with 13B active params from scratch using torchtitan, achieving a smooth loss after 17 trillion tokens and one month on 2,000 B300 GPUs. Details include a low sparsity activation (4/256 experts), using FSDP/HSDP with 120-GPU groups and EP8 inside replicas, a custom load‑balancing algorithm to avoid expert collapse, frequent B300 GPU crashes (~one every 4 hours early on) and 300 restarts handled via a manual replace-and-restart workflow. Results: faster inference than comparable frontier models and stable final training.
AI-written summary · about
chapters (9)
- 0:00Pre-training 400B model
- 1:05Training on 2000 B300 GPUs
- 2:31Inference performance & architecture
- 3:13MoE sparsity & parallelism
- 4:36B300 setup & environment
- 6:00Handling GPU crashes
- 7:24Pre-training complexities
- 8:28Conclusion & open-source
- 9:29Q&A on GPU & framework
related talks
30mOur Journey With TorchTitan
13mLightning Talk: Exploring PiPPY, Tensor Parallel and Torchserve for Large...
26mEfficient MoE Pre-training at Scale on AMD GPUs With TorchTitan
11mUnlock the Future of Generative AI: TorchTitan's Latest...
24mEfficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM
13mFrom Silicon to Scale: Training at Scale
12mThe Ultra-Scale Talk: Scaling Training to Thousands of GPUs
25mDistributed Training in PyTorch: Zero to Hero