Daydream: Accurately Estimating the Efficacy of Optimizations for DNN Training
summary
Daydream builds a fine-grained dependency graph from CUPTI traces plus per-layer timestamps, applies simple graph-transformation primitives (select, shrink, insert/remove, reschedule) to model optimizations like AMP, kernel fusion, batch-norm reconstruction and P3, then simulates execution to predict runtime. It maps kernels to layers with a synchronization-free mapping, predicts distributed and single-GPU changes from one profile, and reports low errors (about 8% average error, up to ~15% max) across five models and several optimizations.
AI-written summary · about
chapters (19)
- 0:00Intro
- 0:30Executive Summary
- 2:30Advances in ML Full Stack Research
- 3:21What-if Questions
- 4:15Why Dependency Analysis
- 5:14Challenges for Dependency Graph Analysis the ML context
- 7:12Daydream Overview
- 8:13Tracking Dependencies
- 10:13Daydream's Graph Construction (cont.)
- 11:10Challenge 2: Trace-Layer Correlation
- 12:21Daydream's Kernel-Layer Mapping
- 13:16Optimization Diversity Technique Examples
- 15:38Daydream's Transformation Primitives (cont.)
- 17:14Example - Automatic Mixed Precision
- 19:58Methodology (cont.)
- 20:25Runtime Estimation Accuracy
- 21:37Estimating Distributed Training
- 22:47Estimating Efficacy of P3
- 23:48Conclusion
related talks
13mOptimizing Dynamic Neural Networks with Brainstorm
15mEffectively Scheduling Computational Graphs of Deep Neural Networks toward Their...
01h 02mMulti-Dimension Mapping of Embedded Applications
22mCentimani: Enabling Fast AI Accelerator Selection for DNN Training with a Novel...
15mIdentifying and Analyzing Pitfalls in GNN Systems
01h 16mOptimizing workloads with Transformers and Diffusers
19mOffloading-Efficient Sparse AI Systems
18mTierTrain: Proactive Memory Tiering for CPU-Based DNN Training