DualPipe from Scratch: Implementing DeepSeek's 5D Parallelism in PyTorch

on YouTube ↗20m

" summary

Implementation-first walkthrough of a PyTorch DualPipe reference that reproduces DeepSeek V3 at 2,048+ GPUs and explains the missing details: exact warm-up formula warmup = floor(P/2) + 2(M-1) that yields 3% bubble overhead and avoids deadlocks; why K_PE is shared across heads* in decoupled RoPE; bias update must run before routing and EMA init = 1/num_experts; sigmoid routing separates selection from gate weights; fix causal-mask offsets; hierarchical all-to-all halves inter-node traffic. Code and tests on github.com/DevJadhav/deepseek-from-scratch.

AI-written summary · about

" chapters (17)

1. 0:00DualPipe introduction
2. 1:03What is DualPipe
3. 4:36DualPipe in PyTorch
4. 6:02Warm-up formula
5. 7:25PyTorch schedule
6. 8:30Async communication
7. 9:32First silent bug
8. 10:37Second silent bug
9. 12:02Third silent bug
10. 12:43Tensor parallelism
11. 13:26Hierarchical all-to-all
12. 14:28Fourth silent bug
13. 15:32Fifth silent bug
14. 16:13Three abstractions
15. 17:19Benchmark results
16. 18:25Why DualPipe matters
17. 19:28Q&A

" related talks

• 20mHetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters throughSam H. Noh, Jay H. Park, Gyeongchan Yun, Chang M. Yi, Nguyen T. Nguyen, Seungmin Lee, Jaesik Choi, Young-ri Choi · USENIX ATC '20
• 16mRevisiting Pipeline Parallelism for LLM ServingSoonjae Hwang, Jeongseob Ahn · 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI '26)
• 44mApropos DeepSeekDan Bergh Johnsson · AI-fokus 2025
• 15mPipeThreader: Software-Defined Pipelining for Efficient DNN ExecutionFeiyang Chen, Lei Wang, Fan Yang, Yining Shi, Yang Wang, Mao Yang, Lingxiao Ma, Zhi Yang, Jilong Xue, Yuqing Xia, Yu Cheng, Zhiwen Mo · OSDI '25
• 17mPPipe: Efficient Video Analytics Serving on Heterogeneous GPU Clusters via...Y. Charlie Hu, Qiang Xu, Z. Jonny Kong · USENIX ATC '25
• 14mFlexPipe: Maximizing Training Efficiency for Transformer-based Models with...Hairui Zhao, Qi Tian, Hongliang Li, Zizhong Chen · USENIX ATC '25
• 29mAligning DeepSeek R1Richard Brough · DevBcn 2025
• 14mFine-tuning giant neural networks on commodity hardware with automatic pipeline...Assaf Schuster, Mark Silberstein, Saar Eliad, Ido Hakimi, Alon De Jagger · USENIX ATC '21