Check-N-Run: a Checkpointing System for Training Deep Learning Recommendation Models

on YouTube ↗16m

" summary

System implements a decoupled, atomic checkpoint pipeline that copies GPU state to CPU and uploads with dedicated CPU workers, targets terabyte-scale recommendation models whose embeddings are >99% of size, and combines differential checkpointing (one‑shot, consecutive, intermittent) with row‑wise quantization (gradient‑tuned ranges). Evaluation on Facebook workloads shows write bandwidth drops 6–17× and storage 2.5–8×; 3‑bit quantization tolerates up to ~3 resumes under a 0.01 accuracy loss threshold before falling back to 8‑bit.

AI-written summary · about

" chapters (15)

1. 0:00Intro
2. 0:30Recommendation Models are important . Use cases include
3. 1:00Recommendation Model Architecture
4. 1:48High Performance Training at Meta
5. 2:26The Criticality of Checkpointing • Failure recovery ensure progress
6. 3:41Checkpoint Challenges
7. 4:52Check-n-Run
8. 6:20Checkpointing Workflow
9. 7:18Reducing WB with Differential Checkpointing
10. 8:09Approaches for Differential Checkpointing • One-Shot Differential Checkpoint . Consecutive Incremental Checkpoint - Intermittent Differential Checkpoint
11. 10:00Checkpoint Quantization Compress checkpoint without degrading training accuracy
12. 11:31Comparing Quantization Strategies . Uniform quantization . Non-uniform quantization using kmeans • Adaptive uniform quantization
13. 12:47Quantization Bit-width Selection
14. 14:35Overall Reduction
15. 15:06Summary

" related talks

• 15mUniversal Checkpointing: A Flexible and Efficient Distributed Checkpointing...Masahiro Tanaka, Olatunji Ruwase, Minjia Zhang, Xinyu Lian, Sam Ade Jacobs, Lev Kurilenko, Stas Bekman · USENIX ATC '25
• 17mCheckmate: Zero Performance Overhead Model Checkpointing via Network Gradient ReplicationManya Ghobadi, Adam Belay, Ankit Bhardwaj, Weiyang Wang, Jeremy Carin · NSDI '26
• 13mByteCheckpoint: A Unified Checkpointing System for Large Foundation Model DevelopmentXin Liu, Haibin Lin, Yanghua Peng, Chuan Wu, Zuquan Song, Borui Wan, Mingji Han, Yiyao Sheng, Mofan Zhang, Zhichao Lai, Menghan Yu, Junda Zhang · NSDI '25
• 12mCheckFreq: Frequent, Fine-Grained DNN CheckpointingVijay Chidambaram, Amar Phanishayee, Jayashree Mohan · FAST '21
• 15mDistributed CheckpointIris Zhang, Chien-Chin Huang · PyTorch Conference 2023
• 15mAdaCheck: An Adaptive Checkpointing System for Efficient LLM Training with Redundancy...Kai Lu, Shengwei Li, Qiaoling Chen, Zhiquan Lai, Peng Sun, Dongsheng Li, Weijie Liu, Keshi Ge · FAST '26
• 32mEfficient Transparent Checkpointing of AI/ML Workloads in Kub...V Spišaková, R. Stoyanov, A. Reber · KubeCon + CloudNativeCon Europe 2025
• 12mQuickUpdate: a Real-Time Personalization System for Large-Scale Recommendation ModelsAssaf Eisenman, Kiran Kumar Matam, Hani Ramezani, Fan Wang, Zeliang Chen, Yue Dong, Maomao Ding, Zhiwei Zhao, Zhengyu Zhang, Ellie Wen · 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI '24)