Fault Tolerance Training at Pinterest
summary
Pinterest implements iteration-level checkpointing using StatefulDataloader and stores model, optimizer and LR scheduler state plus the exact dataloader position so jobs resume without skipping or duplicating samples; integrated into a managed platform with CLI, UI and workflow hooks, reducing thousands of monthly job failures. Ray-based loaders use a virtual epoch split with ~0.8% overhead per subset (10 subsets, 12h epochs); PyTorch loader overhead ~0.3%. PyArrow IPC plus S3 yields ~500x upload speed and ~300x checkpoint size reduction, and users (OMI Sage, PinClip, ranking) resume smoothly.
AI-written summary · about
chapters (18)
- 0:00Fault tolerance at Pinterest
- 3:13Why fault tolerance matters
- 4:59Opportunistic GPU usage
- 5:21Mid-epoch checkpointing challenges
- 7:07Stable PyTorch data loader
- 8:34Ray data loader checkpointing
- 10:25Streaming loader state management
- 11:49Checkpoint serialization optimization
- 13:51Virtual epoch for Ray loaders
- 15:51Seamless checkpoint manager
- 17:18One-line adoption
- 18:00Operator-level fault tolerance
- 18:43Early results and use cases
- 19:25Overhead of fault tolerance
- 20:27End-to-end user experience
- 21:11Diagnosis pipeline and retry UI
- 21:33Future work and performance
- 23:04Q&A: Checkpointing states
related talks
27mPinterest's ML Evolution: Distributed Training with Ray
37mTransparent, Infra-Level Checkpoint and Restore for Resilient AI/ML Workloads
24mThe Evolving Landscape of Dataloading
15mDistributed Checkpoint
34mThe Synthetic Data Flywheel
28mPer-epoch Shuffling Data Loader: Mix It Up As You Train!
33mOvercoming Distributed ML Challenges with Ray Train
25mTraining Through Failures: How Meta Keeps 100k-GPU Jobs Alive With Open-Sou...