ModelKeeper: Accelerating DNN Training via Automated Training Warmup
summary
ModelKeeper automatically initializes new DNN training by transforming weights from previously trained models in a shared cluster, using structure-aware matching (dynamic programming on tensor graphs) and width and depth operators that replicate or insert identity tensors to preserve function. It runs as a cluster service with k‑means clustering and bucketing to scale, evaluated on an 80‑machine cluster across CV and NLP workloads, and yields 1.3×–4.3× faster completion and up to 75% GPU‑hour savings with no final accuracy loss.
AI-written summary · about
chapters (8)
- 0:00Introduction and motivation
- 2:31Repurposing model weights
- 4:36Model keeper architecture
- 5:11System design components
- 6:30Optimization and scaling
- 11:49Implementation details
- 12:24Evaluation and results
- 13:59Conclusion
related talks
17mModel-Switching: Dealing with Fluctuating Workloads in Machine-Learning-as-a-Service
17mTrainMover: An Interruption-Resilient Runtime for ML Training
22mCentimani: Enabling Fast AI Accelerator Selection for DNN Training with a Novel...
18mDISTMM: Accelerating Distributed Multimodal Model Training
42mAlex Smola, Professor in the Machine Learning Department, Carnegie Mellon University
32mTaming the Machine: Basics of ML Models Training and Inference Optimization
18mTierTrain: Proactive Memory Tiering for CPU-Based DNN Training
21mCost Effectively Deploy Thousands of Fine Tuned Gen AI Models Like...