The Ultra-Scale Talk: Scaling Training to Thousands of GPUs
summary
Demonstrates a practical 5D recipe—data, tensor, context, pipeline and expert parallelism—and a running stack that uses nanotron plus PyTorch DDP/FSDP patterns to overlap compute and comms. Claims a global-batch sweet spot near 10 million tokens and shows how naive data-parallel at 32,000 GPUs overshoots that limit. Explains FSDP prefetching, tensor-parallel math split, pipeline scheduling from Deepseek to remove bubbles, and cites real memory limits (very large models that won’t fit one GPU) when scaling to thousands of GPUs.
AI-written summary · about
chapters (12)
- 0:00Ultra-scale training intro
- 1:04GPU cluster architecture
- 2:28Single-GPU training
- 2:49Data parallelism overhead
- 4:16Global batch size limits
- 5:21FSDP and model size
- 7:08Tensor parallelism
- 8:33Context parallelism
- 9:39Pipeline parallelism
- 10:43Mixture-of-experts parallelism
- 11:465D parallelism cheat sheet
- 12:07Energy efficiency
related talks
13mFrom Silicon to Scale: Training at Scale
24mTraining large scale models using PyTorch
21mLessons Learned in WatsonX Training: Scaling Cloud-Native...
10mKeynote: How to Leverage PyTorch to Scale AI Training and Inferencing
16mMegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs
34mDeep Learning at Scale
10mLightning Talk: Scaling Recommendation Systems To 2K GPUs and Beyond
08mPyTorch Distributed: Towards Large Scale Training