Obscura: Concealing Recomputation Overhead in Training of Large Language Models...
summary
Obscura is a pipeline training system that transforms the standard 1F1B schedule to shift forward work into unused forward bubbles and turn them into backward bubbles, integrates swapping-aware recomputation and solves an optimization problem to pick which operators to recompute under memory and bandwidth constraints. It adds a partition-adjustment algorithm to rebalance layers, runs on 80 GB NVIDIA GPUs with DeepSpeed, and shows up to 1.33× throughput improvement on Llama-2 and GPT-3 scale models.
AI-written summary · about
chapters (11)
- 0:00Obscura introduction
- 1:45Pipeline parallelism challenges
- 3:30Recomputation trade-offs
- 6:22Strawman pipeline
- 9:22Obscura system overview
- 10:52Dependency relaxation
- 12:41Swapping-aware recomputation
- 15:09Partition adjustment
- 16:18Evaluation setup
- 17:53Performance results
- 19:26Conclusion
related talks
23mOperationalizing Large Language Model Training Pipelines
14mScalable Solutions for Running Large Language Models
16mWLB-LLM: Workload-Balanced 4D Parallelism for Large Language Model Training
41mSub-Linear Algorithms meets Large Language Models
19mSystem for Scaling Large Language Models
30mUnderstanding the end-to-end LLM training and inference pipeline
16mRevisiting Pipeline Parallelism for LLM Serving
22mPerformance Optimizations at 100K+ Scale