Predicting and optimizing the behavior of large ML models
summary
Presents a concrete method that differentiates the final model performance with respect to pre‑training choices by computing a meta gradient through training (using deterministic SGD and recursive rematerialization to get roughly log T memory and T log T cost). Computing the meta gradient costs about 3–5× a single training run; ~40 meta‑steps finds data subsets that improve a leaderboard metric by ≈4 percentage points; inserting 2% poisoned examples drops CIFAR accuracy by ~14%.
AI-written summary · about
chapters (9)
- 0:00Introduction and motivation
- 4:15The meta parameter framework
- 14:01Metagradient approach
- 20:02Applications in data selection
- 28:31Data poisoning experiments
- 35:01Predictive modeling
- 40:05Computational challenges
- 50:10Optimization smoothness
- 59:43Conclusion and future work
related talks
46mIncreasing Participant Agency by Understanding the Impact of Training Data
50mFlexible Neural Networks and the Frontiers of Meta-Learning
47mMeta-learning of Optimizers and Update Rules
23mTTT, TAO, δ-mem: When to Spend a Gradient
32mTaming the Machine: Basics of ML Models Training and Inference Optimization
41mOptimization Algorithms for Deep Learning
33mTradeoffs in Large Scale Learning: Statistical Accuracy vs. Numerical Precision
21mOperationalizing ML Training Infra at Meta Scale