Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNs
summary
Bamboo makes preemptible (spot) instances usable for large DNN training by building a distributed system that inserts redundant computations into idle pipeline bubbles of pipeline-parallel training, placing replicas across availability zones to avoid correlated losses. It scales pipeline depth ≈1.5×, uses a simulator plus real spot traces and runs BERT-large experiments, and claims 3.7× throughput over checkpointing and 2.4× cost reduction versus on‑demand while open‑sourcing the implementation.
AI-written summary · about
chapters (17)
- 0:00Intro
- 0:28Generative Al Is Changing the World
- 1:16Model Sizes are Increasing
- 1:52Prohibitive Costs for Most Organizations
- 3:29Spot Instances Can Lower Costs
- 4:23Spot Instances Can Have High Failure Rates
- 6:17Pipeline Parallelism Enables Large Model Training
- 6:36Redundancy Provides Resilience
- 7:32Pipeline Parallelism Has Bubbles
- 8:10Pipeline Bubbles Still Exist
- 8:52Consecutive Failures Are Fatal
- 10:29Evaluated Bamboo On Many Datasets
- 11:00Experiments
- 11:39We Provide Comparable Performance to On-Demand
- 12:09Bamboo Significantly Reduces Cost
- 12:43Simulation of BERT to Completion
- 14:42Bamboo Provides Resilience on Preemptible GPUs
related talks
16mParcae: Proactive, Liveput-Optimized DNN Training on Preemptible Instances
25mAccelerate NLP training with Amazon SageMaker
17mAttack of the Bubbles: Straggler-Resilient Pipeline Parallelism for Large Model Training
24mFast training and near-linear scaling with DataParallel in Amazon SageMaker
27mFoundation Models: Blueprints for success
27mDeveloping and Deploying Models at Scale — without Spark!
29mTrain billion-parameter models with model parallelism on Amazon SageMaker
29mRunning Massively Parallel Deep-learning Inference Pipelines on Kub... Suneeta Mall & Martin Abeleda