Speeding Up LLM Training by Over 45%
summary
Describes bottlenecks in training large language models on big clusters and presents practical solutions: the YaFSDP library to reduce communication inefficiencies, memory optimizations, and use of 3D/4D parallelism for scaling to very large models; also discusses techniques used to speed up Mixture-of-Experts (MoE) training.
AI-written summary · about
chapters (8)
- 0:00Introduction to faster training
- 7:18GPU and memory profiling
- 13:41Training 7B model
- 22:13Activation issues and checkpointing
- 30:43Training 35B model
- 33:35Tensor parallelism and future
- 35:47Practical optimizations and summary
- 38:19Q&A
related talks
31mVeRL: Extreme Optimization Practices for Ultra-Large-Scale MoE Models
19mAccelerating the Training of Large Language Models using Efficient Activation...
30mLearning to wield large language models
14mScalable Solutions for Running Large Language Models
23mOperationalizing Large Language Model Training Pipelines
08mHacks to Make LLM Training Faster
16mWLB-LLM: Workload-Balanced 4D Parallelism for Large Language Model Training
19mSystem for Scaling Large Language Models