Universal Checkpointing: A Flexible and Efficient Distributed Checkpointing...
summary
Universal Checkpointing decouples on-disk state from model parallel layout by converting checkpoints into an atomic checkpoint format that stores parameters plus full optimizer context, then remaps them via a pattern-based reconfiguration pipeline (tensor/pipeline/data/ZeRO/sequence, plus sharding hybrids). A nested-parallel conversion engine uses node-level CPU parallelism to cut reconfiguration time to under three minutes for 1 TB models and adds ≈0.01% training overhead. Implemented in DeepSpeed, validated on thousands of GPUs (NVIDIA/AMD) and used in real LLM training at Microsoft and BigScience.
AI-written summary · about
chapters (8)
- 0:00Universal Checkpointing intro
- 1:24Distributed checkpointing challenges
- 4:13Universal Checkpointing System
- 7:24Atomic checkpoint representation
- 8:50Pattern-Based reconfiguration
- 10:13Efficiency optimizations
- 12:01Experimental validation
- 14:31Conclusion and future
related talks
13mByteCheckpoint: A Unified Checkpointing System for Large Foundation Model Development
15mDistributed Checkpoint
16mCheck-N-Run: a Checkpointing System for Training Deep Learning Recommendation Models
10mLightning Talk: In-Cluster Distributed Checkpointing: Optimizing Training...
17mCheckmate: Zero Performance Overhead Model Checkpointing via Network Gradient Replication
15mAdaCheck: An Adaptive Checkpointing System for Efficient LLM Training with Redundancy...
32mEfficient Transparent Checkpointing of AI/ML Workloads in Kub...
12mCheckFreq: Frequent, Fine-Grained DNN Checkpointing