Analyzing Optimization and Generalization in Deep Learning via Trajectories of Gradient Descent
summary
Analyzes gradient flow and gradient descent over deep linear nets to get concrete dynamics: balancedness preserved at common initializations and an explicit PSD preconditioner (closed form) so the end-to-end matrix follows modified gradient dynamics that promote directions already taken. Proves linear-rate (exponential) convergence for full-batch GD under two mild initialization conditions (output-dim=1 holds with ~0.25 probability), shows depth can accelerate GD for P>2 losses (experiments with L4), and demonstrates in matrix completion that deeper factorized models produce larger singular-value gaps and lower effective rank where nuclear-norm minimization fails.
AI-written summary · about
chapters (9)
- 0:00Introduction and motivation
- 1:06Classical vs. deep learning
- 3:35Deep linear neural networks
- 5:51Gradient flow and dynamics
- 10:54Optimization results
- 20:30The role of depth in training
- 25:16Implicit regularization
- 36:06Singular value dynamics
- 41:28Conclusion and future work
related talks
41mOptimization Algorithms for Deep Learning
11mOn Implicit Regularization in Deep Learning
01h 01mOptimization's Untold Gift to Learning: Implicit Regularization
23mOn Implicit Regularization in Deep Learning
13mThe dynamical view of machine learning: implicit biases of large learning rates
01h 20mThe elusive generalization: classical bounds to double descent to grokking
01h 18mOptimization I
58mFeature Selection with Gradient Descent on Two-layer Networks in Low-rotation Regimes