Distributed Training in PyTorch: Zero to Hero
summary
A step-by-step GitHub guide shows how to scale a single-GPU script to multi-node production, with runnable examples and diagnostic tips. Practical claims: train a 405B-sized model with pure PyTorch, add ZeRO as a two-line change to cut optimizer memory (example: Llama 8B moves from OOM to ~60 GB with ZeRO), and then use PyTorch FSDP to shard weights down to ~40 GB. Notes: a 45B run used 64 H100s at ~30s/iter and Llama 4f required ~16,000 GPUs; pipeline/tensor/context parallelism reduce costly all-gather overhead.
AI-written summary · about
chapters (20)
- 0:00Distributed training intro
- 1:04Why use distributed training
- 2:27Data parallelism basics DDP
- 3:50PyTorch DDP under the hood
- 4:33Live demo single GPU
- 5:38Live demo distributed data parallel
- 6:45DDP optimizer memory issues
- 7:08Zero memory optimization DDP
- 8:12Fully sharded data parallel
- 10:18FSDP2 simplified API
- 11:22Live demo Zero vs FSDP memory
- 12:25Multi-node training FSDP
- 13:48Live demo multi-node FSDP
- 14:29Challenges 4B+ models
- 15:14Introducing 4D parallelism
- 16:18Other parallelism techniques
- 17:10Q&A FSDP parallelism
- 20:03Q&A FSDP overhead
- 22:32Q&A multi-node IP config
- 23:35Q&A tensor parallel
related talks
14mZeRO-Offload: Democratizing Billion-Scale Model Training
24mTraining large scale models using PyTorch
12mThe Ultra-Scale Talk: Scaling Training to Thousands of GPUs
38mMPI Meets Machine Learning: Unlocking PyTorch distributed for scaling AI workloads
21mLessons Learned in WatsonX Training: Scaling Cloud-Native...
13mFrom Silicon to Scale: Training at Scale
08mPyTorch Distributed: Towards Large Scale Training
10mKeynote: How to Leverage PyTorch to Scale AI Training and Inferencing