DeepSpeed – Efficient Training Scalability for Deep Learning Models
summary
DeepSpeed presents concrete systems: Zero optimizer shards model state to remove GPU memory limits and enables training from a single GPU up to 100k GPUs; universal checkpointing decouples checkpoints from hardware so a GPT-176B run resumes on half the GPUs mid-job; Domino communication hiding overlaps tensor-parallel comm and compute, showing ~16% end-to-end speedup and full hide on H21/H200 single-node profiles with 75% hiding in current multi-node tests; DeepCompile auto-shards and profiles to give up to 50% speedups (O3), 5x for O3+offload, and NVMe IO work reaches ~50 GB/s reads, 25 GB/s writes and 20x faster checkpointing.
AI-written summary · about
chapters (12)
- 0:00DeepSpeed introduction and goals
- 1:04Why DeepSpeed was created
- 3:54GPU memory wall problem
- 6:47Recent DeepSpeed advancements
- 7:09Universal checkpoint flexibility
- 9:56Parallelism strategy reconfiguration
- 10:37Communication-free training with Domino
- 12:43DeepSpeed compiler optimizations
- 14:49Automating sharding and prefetching
- 16:34IO bottlenecks optimization
- 17:38Storage innovations and gains
- 18:00Community adoption and future
related talks
26mDeepSpeed: Efficient Training Scalability for Deep Learning
10mvLLM & Deepspeed Updates
26mEfficiently serving LLMs at scale.
12mFlashNeuron: SSD-Enabled Large-Batch Training of Very Deep Neural Networks
49mDistributed Deep Learning
17mAccelerating Collective Communication in Data Parallel Training across Deep Learning...
36mAccelerating High-Performance Machine Learning at Scale i...
25mOpen Source Model Performance Optimization With SGLang