Kernel and Deep Regimes in Overparameterized Learning
summary
Gradient descent on overparameterized models converges to qualitatively different minimizers depending on parameterization and initialization scale: factorized matrix estimation with tiny initialization and small step sizes yields minimum nuclear norm solutions and good generalization, while large-scale initialization lands in the neural tangent kernel regime (minimum RKHS/L2 bias). A squared-parameter linear regression interpolates between minimum L1 norm and L2 solutions via an explicit “hyperentropy” complexity; logistic/exponential losses instead drive max-margin behavior.
AI-written summary · about
chapters (17)
- 0:00Intro
- 0:31Deep learning
- 2:52Gradient descent learning
- 9:44Optimization bias in matrix factorization
- 12:16Rich inductive bias implicitly from gradient descent
- 16:07Neural Tangent Kernels
- 19:21Optimization bias in kernel regime
- 20:37NTK regime in homogeneous models
- 24:21Kernel vs deep regime in matrix completion
- 25:45Kernel vs deep regimes
- 28:09Linear regression with squared parameterization
- 29:39Optimization bias with squared parameterization
- 33:16Kernel - deep regime transition
- 34:48Sparse learning
- 36:31Relationship to explicit regularization
- 39:19Logistic loss on separable dataset
- 42:58Concluding remarks
related talks
01h 23mImplicit Regularization II
01h 17mImplicit Regularization I
11mOn Implicit Regularization in Deep Learning
01h 01mOptimization's Untold Gift to Learning: Implicit Regularization
13mThe dynamical view of machine learning: implicit biases of large learning rates
01h 14mRecent Developments in Over-parametrized Neural Networks, Part II
29mImplicit Regularization in Nonconvex Statistical Estimation
13mNon-Parametric Convergence Rates for Plain Vanilla Stochastic Gradient Descent