Training large scale models using PyTorch
summary
Covers concrete methods and tradeoffs for scaling: using NCCL with distributed data parallel to double batch throughput when adding GPUs, pipeline parallelism to reduce idle time, and tensor parallelism to shard single huge layers across devices. Memory tactics include CPU offloading, activation checkpointing and mixed precision, plus fully sharded data parallel (FSDP) to shard params gradients and optimizer state. Notes torch.distributed primitives, elastic recovery, and active use of FSDP for multimodal foundation model training.
AI-written summary · about
chapters (8)
- 0:00Training large scale models importance
- 1:49Scaling challenges and PyTorch solutions
- 3:11Distributed data parallel in PyTorch
- 7:06Model parallelism for large models
- 13:09Tensor parallelism for massive layers
- 16:43Memory reduction techniques
- 21:17Advanced PyTorch scaling features
- 21:59Q&A
related talks
08mPyTorch Distributed: Towards Large Scale Training
27mTips and tricks for distributed large model training
38mMPI Meets Machine Learning: Unlocking PyTorch distributed for scaling AI workloads
12mThe Ultra-Scale Talk: Scaling Training to Thousands of GPUs
21mLessons Learned in WatsonX Training: Scaling Cloud-Native...
14mLightning Talk: Large-Scale Distributed Training with Dynamo and...
25mDistributed Training in PyTorch: Zero to Hero
34mSpeeding up Your ML Workload: Pytorch Compile and Distributed Training