Fine-tuning giant neural networks on commodity hardware with automatic pipeline...
summary
FTPipe presents a practical pipeline model‑parallel system that fine‑tunes giant transformers (BERT‑340M, GPT2‑1.5B, T5‑3B) on commodity PCIe GPUs by partitioning models into arbitrary computational blocks with a Mixed‑pipe allocator that allows non‑adjacent blocks on the same GPU. It runs on 8 RTX2080‑Ti with 11GB each, overlaps communication and computation, cuts T5 3B pipeline communication to ~486MB versus 12GB for data‑parallel, uses recomputation and optional weight‑stashing, and achieves up to 3× speedup with no measurable accuracy loss.
AI-written summary · about
chapters (7)
- 0:00Introduction to fine-tuning
- 1:20Existing pipeline frameworks
- 2:49Limitations of partitioning
- 5:34Introducing MixedPipe
- 7:15The partitioning algorithm
- 9:23Implementation details
- 11:51Experimental results
related talks
14mFlexPipe: Maximizing Training Efficiency for Transformer-based Models with...
17mPPipe: Efficient Video Analytics Serving on Heterogeneous GPU Clusters via...
16mRevisiting Pipeline Parallelism for LLM Serving
20mHetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through
13mLightning Talk: Exploring PiPPY, Tensor Parallel and Torchserve for Large...
13mFrom Silicon to Scale: Training at Scale
24mEfficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM
40mOptimize Your AI Cloud Infrastructure: A Hardware Perspective