Accelerating the Training of Large Language Models using Efficient Activation...
summary
Describes two activation rematerialization techniques—Pipeline-Parallel-Aware Offloading (uses host memory to store activations) and Compute-Memory Balanced Checkpointing (trades off activation memory and compute)—plus an efficient search method to pick optimal hybrid-parallelism settings that combine offloading and checkpointing. Evaluates these techniques on public LLM benchmarks (including a 175B Llama-like model with 32,768 context on 256 NVIDIA H800) and reports improvements in Model FLOPs Utilization.
AI-written summary · about
related talks
16mWLB-LLM: Workload-Balanced 4D Parallelism for Large Language Model Training
46mSpeeding Up LLM Training by Over 45%
19mNo GPU Left Behind: Scaling Online LLM Training With...
30mLearning to wield large language models
08mHacks to Make LLM Training Faster
38mUnraveling Multimodality with Large Language Models
15mOptimus: Accelerating Large-Scale Multi-Modal LLM Training by Bubble Exploitation
22mTraining a LLaMA in your Backyard: Fine-tuning Very Large...