The dynamical view of machine learning: implicit biases of large learning rates

on YouTube ↗13m

" summary

Large learning rates cause gradient descent to escape sharp minima toward flatter ones and produce edge of stability behavior where limiting sharpness converges to about 2/H. A new dynamical convergence analysis (nonconvex, no Lipschitz gradient) shows on two‑layer linear models the training dynamics impose an implicit nuclear-norm regularizer with layer norm differences controlled by roughly 2/H. For functions with very high growth, dynamics instead yield a one-sided stability 1/H bound.

AI-written summary · about

" chapters (10)

1. 0:00Dynamical systems in ML
2. 1:00Dynamical systems for optimization
3. 2:24Implicit biases of large learning rates
4. 3:05Gradient descent with large learning rates
5. 4:56Large learning rates and flat minima
6. 6:41Edge of stability
7. 7:46Implicit nuclear norm regularization
8. 8:50Regularity for stability
9. 9:34Future directions
10. 10:36Q&A

" related talks

• 05mNon-monotonic convergence of gradient descent with large stepsizeJingfeng Wu · 9th Annual Simons Institute Industry Day
• 11mOn Implicit Regularization in Deep LearningWei Hu · Meet the Fellows Welcome Event
• 46mAnalyzing Optimization and Generalization in Deep Learning via Trajectories of Gradient DescentNadav Cohen · Frontiers of Deep Learning 2019
• 46mKernel and Deep Regimes in Overparameterized LearningSuriya Gunasekar · Frontiers of Deep Learning 2019
• 01h 23mImplicit Regularization IINati Srebro · Deep Learning Boot Camp 2019
• 01h 01mOptimization's Untold Gift to Learning: Implicit RegularizationNathan Srebro Bartom · Optimization, Statistics and Uncertainty
• 59mTutorial: Statistical Learning Theory and Neural Networks ISpencer Frei · Deep Learning Theory Workshop and Summer School 2022
• 01h 18mOptimization IBen Recht · Big Data Boot Camp 2013