Learning and Generalization in Over-parametrized Neural Networks, Going Beyond Kernels
summary
Shows that under OPS condition (very large width, careful random init, tiny learning rate) SGD on deep ReLU nets provably converges to the neural tangent kernel solution, so training is essentially linear. Breaking those assumptions—milder over-parameterization or large noisy steps or a reparameterized last layer—lets a three‑layer ResNet learn compositional targets with sample/time ~O((C_f + C_g)/α^4) while any kernel must pay for the composite complexity (examples up to exponential in D in constructions); experiments match the separation, outperforming NTK on CIFAR‑style tasks.
AI-written summary · about
chapters (19)
- 0:00Introduction
- 7:50Tangent Kernel
- 9:18Theorem
- 13:23Intuition
- 15:56Takeaway
- 16:34Why Neural Networks
- 17:55Results
- 19:05Learning Network
- 19:42Concept Class
- 24:17Learning Algorithms
- 26:20Generalization Theorem
- 31:32Complexity
- 34:32Extensions
- 39:17Intuitions
- 40:37Higher Level Intuition
- 41:52Key Message
- 44:15Feed Better
- 46:35Neural Network
- 47:33Future Direction
related talks
01h 14mRecent Developments in Over-parametrized Neural Networks, Part I
01h 14mRecent Developments in Over-parametrized Neural Networks, Part II
01h 04mNeural Networks meet Nonparametric Regression: Generalization by Weight Decay and Large...
59mTutorial: Statistical Learning Theory and Neural Networks I
19mTraining (Overparametrized) Neural Networks in Near-Linear Time
32mRobust Learning of a Single Neuron via Sharpness
01h 04mRandomized Embeddings and Neural Networks
56mHow neural networks learns simple functions?