Whale: Efficient Giant Model Training over Heterogeneous GPUs
summary
Whale is a distributed training framework that adds two model-annotation primitives, replicate and split, to transform a local DAG into a distributed plan and insert bridge layers automatically. It implements a hardware-aware balancing algorithm for heterogeneous clusters (V100/P100 mixes), achieves up to ~1.45x speedup over GPipe and 1.14x in other cases, hybrid gains of 1.2–2.4x on classification, about 20% pipeline speedup, and scales a multimodal M6 model to 10 trillion parameters on 512 V100 GPUs with minimal code changes.
AI-written summary · about
chapters (10)
- 0:00Challenges in large models
- 1:22Parallel training strategies
- 3:30Heterogeneous gpu constraints
- 5:45Proposed framework overview
- 7:04Task graphs and primitives
- 9:44Hardware-aware load balancing
- 12:18Balancing strategies
- 13:20Performance evaluations
- 17:19Impact of load balancing
- 18:43Scaling and conclusion
related talks
20mHetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through
27mTips and tricks for distributed large model training
31mThe Evolution of Multi-GPU Inference in vLLM
14mFine-tuning giant neural networks on commodity hardware with automatic pipeline...
24mEfficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM
26mDeepSpeed: Efficient Training Scalability for Deep Learning
17mCharacterization of Large Language Model Development in the Datacenter
16mMegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs