HetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through

on YouTube ↗20m

" summary

HetPipe is a system that combines pipelined model parallelism with data parallelism by grouping heterogeneous GPUs into a virtual worker that runs concurrent minibatches and synchronizes across workers with Wave Synchronous Parallel. It implements push/pull clocks, proves WSP convergence, and evaluates on a mixed-power GPU cluster: throughput improves 1.4–1.8× in some configs, up to 2.3× when adding whimpy GPUs, and end-to-end training converges up to 49% faster than a state‑of‑the‑art DP baseline (39% faster in a 12→16 GPU experiment).

AI-written summary · about

" chapters (11)

1. 0:00Introduction
2. 0:27Motivation Background
3. 6:40Single Virtual Occur
4. 13:50Evaluation
5. 14:33Partitioning
6. 15:01Equal Distribution
7. 15:26Hybrid Policy
8. 15:56Parameter Placement Policy
9. 16:24Local Placement Policy
10. 18:14Convergence
11. 19:28Conclusion

" related talks

• 17mPPipe: Efficient Video Analytics Serving on Heterogeneous GPU Clusters via...Y. Charlie Hu, Qiang Xu, Z. Jonny Kong · USENIX ATC '25
• 21mWhale: Efficient Giant Model Training over Heterogeneous GPUsWei Lin, Yong Li, Jie Zhang, Langshi Chen, Wencong Xiao, Ziji Shi, Zhen Zheng, Xianyan Jia, Le Jiang, Ang Wang, Xinyuan Li, Xiaoyong Liu, Ang Wang, Xinyuan Li, Xiaoyong Liu · USENIX ATC '22
• 15mPipeThreader: Software-Defined Pipelining for Efficient DNN ExecutionFeiyang Chen, Lei Wang, Fan Yang, Yining Shi, Yang Wang, Mao Yang, Lingxiao Ma, Zhi Yang, Jilong Xue, Yuqing Xia, Yu Cheng, Zhiwen Mo · OSDI '25
• 14mFine-tuning giant neural networks on commodity hardware with automatic pipeline...Assaf Schuster, Mark Silberstein, Saar Eliad, Ido Hakimi, Alon De Jagger · USENIX ATC '21
• 17mAttack of the Bubbles: Straggler-Resilient Pipeline Parallelism for Large Model TrainingWei Wang, Yinghao Yu, Liping Zhang, Tianyuan Wu, Lunxi Cao, Hanfeng Lu, Xiaoxiao Jiang, Siran Yang, Guodong Yang, Jiamang Wang, Lin Qu · NSDI '26
• 21mEnvPipe: Performance-preserving DNN Training Framework for Saving EnergyYoungjin Kwon, Inhoe Koo, Jeongseob Ahn, Myeongjae Jeon, Sangjin Choi · USENIX ATC '23
• 14mFlexPipe: Maximizing Training Efficiency for Transformer-based Models with...Hairui Zhao, Qi Tian, Hongliang Li, Zizhong Chen · USENIX ATC '25
• 16mRevisiting Pipeline Parallelism for LLM ServingSoonjae Hwang, Jeongseob Ahn · 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI '26)