Neural Networks meet Nonparametric Regression: Generalization by Weight Decay and Large...

on YouTube ↗01h 04m

" summary

Claims that deep ReLU models, when written as a parallel deep ReLU architecture and reparameterized via AM‑GM, map weight decay to a total variation‑style penalty; with M > n and depth L ~ log n the estimator attains near‑minimax locally adaptive rates. Theory uses a representer‑type argument and comparisons to trend‑filtering; experiments show very overparameterized nets collapse to few active blocks (≈6) and that weight decay tuning (via holdout) is enough, while large learning rate prevents interpolation and biases toward flatter, generalizing solutions.

AI-written summary · about

" chapters (10)

1. 0:00Introduction
2. 1:04Curve fitting and deep learning
3. 8:26Nonparametric regression
4. 12:57Two-layer ReLU networks
5. 16:30Splines and neural networks
6. 27:29Parallel networks and adaptivity
7. 38:35Experimental results
8. 40:22Optimization and learning rate
9. 55:58Loss landscape and biases
10. 59:59Q&A

" related talks

• 50mLearning and Generalization in Over-parametrized Neural Networks, Going Beyond KernelsYuanzhi Li · Frontiers of Deep Learning 2019
• 46mOn the Foundations of Deep Learning: SGD, Overparametrization, and GeneralizationJason Lee · Frontiers of Deep Learning 2019
• 59mTutorial: Statistical Learning Theory and Neural Networks ISpencer Frei · Deep Learning Theory Workshop and Summer School 2022
• 01h 14mRecent Developments in Over-parametrized Neural Networks, Part IJason Lee · Deep Learning Boot Camp 2019
• 01h 02mTutorial on Deep Learning IRuslan Salakhutdinov · Foundations of Machine Learning Boot Camp 2017
• 11mOn Implicit Regularization in Deep LearningWei Hu · Meet the Fellows Welcome Event
• 46mKernel and Deep Regimes in Overparameterized LearningSuriya Gunasekar · Frontiers of Deep Learning 2019
• 01h 14mRecent Developments in Over-parametrized Neural Networks, Part IIJason Lee · Deep Learning Boot Camp 2019