Attack of the Bubbles: Straggler-Resilient Pipeline Parallelism for Large Model Training
summary
PIPEMORPH is a system that adapts pipeline schedules and offloads communication to host memory with CPU-side RDMA delegation to eliminate head-of-line blocking and absorb delays via pipeline slackness. It detects stragglers, increases warm-up forward count differences between affected stages, and moves comms off GPU. Experiments inject up to 60 ms delays and a 13 ms case, showing iteration time improvements of 1.2–3.5×, and throughput gains of 11% and 36% (rising to 30% and 47% under severe congestion) when training a 140B model.
AI-written summary · about
chapters (15)
- 0:00Pipeline parallelism challenges
- 1:01Computation vs communication stragglers
- 2:25Communication stragglers impact
- 3:31Cascading dependency bubbles
- 4:36Two-stage pipeline delays
- 6:00Pipeline slackness timing
- 7:46Slackness absorbs delays
- 9:11Straggler mitigation algorithm
- 9:53Head-of-line blocking in GPU
- 10:58CPU communication delegation
- 12:29PipeMorph design
- 13:11Evaluation with delays
- 13:53Performance gains
- 14:58Real-world 140B training
- 15:39PipeMorph summary
related talks
20mHetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through
14mFlexPipe: Maximizing Training Efficiency for Transformer-based Models with...
17mPPipe: Efficient Video Analytics Serving on Heterogeneous GPU Clusters via...
19mCrossPipe: Towards Optimal Pipeline Schedules for Cross-Datacenter Training
15mOptimus: Accelerating Large-Scale Multi-Modal LLM Training by Bubble Exploitation
22mCustom Pipeline Parallelism for Frontier-Scale Model Serving
14mFine-tuning giant neural networks on commodity hardware with automatic pipeline...
16mRevisiting Pipeline Parallelism for LLM Serving