Meta-learning of Optimizers and Update Rules
summary
Meta-learning replaces hand-designed optimizers with a parametric per-weight MLP optimizer (≈2k outer parameters) and faces three practical failures: outer-loop cost (inner runs up to 10,000 steps), biased gradients from short unrolls, and a chaotic outer loss. Solutions: variational smoothing (Gaussian-convolution of outer loss) plus annealed unroll length, and combining reparameterization vs ES gradient estimators (RP has ~10^5× lower variance early, ES wins later by ~10^20–10^25). Result: optimizer meta-trained on ImageNet 10-way ConvNets outperforms heavily tuned Adam on validation/test when meta-trained against validation loss; meta-learned unsupervised update rules transfer to IMDB before image-domain overfit.
AI-written summary · about
chapters (11)
- 0:00Start
- 0:45Outline
- 1:23Deep learning
- 6:24Outer-optimization is hard
- 8:23Outer Training Methods
- 9:14Truncated Backpropagation Through Time
- 18:16Outer Loss surface
- 26:34Two unbiased gradient estimators
- 27:34Different variance properties
- 39:29Outer-test example problem
- 44:59Summary
related talks
50mFlexible Neural Networks and the Frontiers of Meta-Learning
01h 04mPredicting and optimizing the behavior of large ML models
46mIncreasing Participant Agency by Understanding the Impact of Training Data
31mUnderstanding Outer Optimizers in Local SGD: Learning Rates, Momentum, and Acceleration
01h 14mRecent Developments in Over-parametrized Neural Networks, Part I
01h 04mNeural Networks meet Nonparametric Regression: Generalization by Weight Decay and Large...
53mOff-policy Policy Optimization
46mGradient Descent: The Ultimate Optimizer