TrainMover: An Interruption-Resilient Runtime for ML Training
Talkat 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI '26) · 2026-07-13..07-15·
summary
TrainMover is an ML training runtime that prepares standby GPUs ahead of interruptions by pre-recording output tensors and running communication-free sandboxed warmup so joiners avoid full initialization. It patches the communication stack to perform a delta-based communication group setup by disconnecting and reconnecting RDMA peers instead of rebuilding groups, and uses a general standby design that maps any role onto spare machines with zero memory overhead. Evaluation shows ~20 seconds downtime at 1024 GPUs (versus ~300 s for Megatron), projecting a 55% GPU-hour savings and 1.4M GPU-hours/week at 64K.
AI-written summary · about
chapters (8)
- 0:00TrainMover project
- 1:02ML training interruptions
- 4:30Handling interruptions today
- 7:39TrainMover’s goal
- 9:05Preparing for interruptions
- 10:10Design challenges and solutions
- 13:43Performance gains
- 15:30Key takeaways
related talks
19mNo GPU Left Behind: Scaling Online LLM Training With...
25mTraining Through Failures: How Meta Keeps 100k-GPU Jobs Alive With Open-Sou...
15mColocating ML Inference and Training with Fast GPU Memory Handover
21mEnhancing Runtime Reliability in LLM Training via Fine-Grained Observability
33mOvercoming Distributed ML Challenges with Ray Train
39mTrain ML models at scale with Amazon SageMaker Training
17mAttack of the Bubbles: Straggler-Resilient Pipeline Parallelism for Large Model Training
18mTierTrain: Proactive Memory Tiering for CPU-Based DNN Training