On the Foundations of Deep Learning: SGD, Overparametrization, and Generalization

on YouTube ↗46m

" summary

Experiment with a 50-neuron teacher and 100-neuron learner shows SGD fails on the well-specified 50-neuron model but hits zero test loss every run after doubling width, so over-parameterization empirically makes optimization easier. Theory proves very wide nets (width M large) keep the Gram/NTK stable so gradient descent converges to an epsilon global minimum; logistic loss dynamics bias towards max‑margin (SVM) solutions; linear deep architectures map L2 weight decay to a Schatten quasi‑norm on the function. Practical caveats: NTK behavior holds for small learning rates and finite‑width/large‑LR models deviate from the kernel regime, so kernel analysis can mispredict state‑of‑the‑art training.

AI-written summary · about

" chapters (27)

1. 0:00Intro
2. 1:32Fundamental Questions
3. 1:49Challenges
4. 2:32What if the Landscape is Bad?
5. 4:58Gradient Descent Finds Global Minima
6. 5:47Idea: Study Dynamics of the Prediction
7. 7:52Local Geometry
8. 8:44Local vs Global Geometry
9. 19:49What about Generalization Error?
10. 20:32Does Overparametrization Hurt Generalization?
11. 21:48Background on Margin Theory
12. 22:40Max Margin via Logistic Loss
13. 23:32Intuition
14. 24:37Overparametrization Improves the Margin
15. 25:04Optimization with Regularizer
16. 29:06Comparison to NTK
17. 31:38Is Regularization Needed?
18. 31:54Warmup: Logistic Regression
19. 32:54What's Special About Gradient Descent?
20. 34:03Changing the Geometry: Steepest Descent
21. 34:35Steepest Descent: Examples
22. 35:05Beyond Linear Models: Deep Networks
23. 36:38Implicit Regularization: NTK vs Asymptotic
24. 38:36Does Architecture Matter?
25. 39:20Example: Changing the Depth in Linear Network
26. 40:24Example: Depth in Linear Convolutional Network
27. 41:52Random Thoughts

" related talks

• 01h 14mRecent Developments in Over-parametrized Neural Networks, Part IJason Lee · Deep Learning Boot Camp 2019
• 01h 14mRecent Developments in Over-parametrized Neural Networks, Part IIJason Lee · Deep Learning Boot Camp 2019
• 01h 37mTutorial: Implicit Bias IINati Srebro · Deep Learning Theory Workshop and Summer School 2022
• 01h 04mNeural Networks meet Nonparametric Regression: Generalization by Weight Decay and Large...Yu-Xiang Wang · Emerging Generalization Settings 2024
• 01h 20mThe elusive generalization: classical bounds to double descent to grokkingMisha Belkin · Modern Paradigms in Generalization Boot Camp 2024
• 01h 02mTutorial: Statistical Learning Theory and Neural Networks IISpencer Frei · Deep Learning Theory Workshop and Summer School 2022
• 01h 21mTutorial: Implicit Bias INati Srebro · Deep Learning Theory Workshop and Summer School 2022
• 01h 01mOptimization's Untold Gift to Learning: Implicit RegularizationNathan Srebro Bartom · Optimization, Statistics and Uncertainty