Computation in Very Wide Neural Networks

on YouTube ↗49m

" summary

Claims that fully connected networks with width → ∞ map to Gaussian processes and that training dynamics linearize to a frozen kernel are supported with formulas and experiments. Bayesian inference for squared loss reduces to exact linear algebra, and NNGP posteriors often match or beat hyperparameter‑tuned SGD on MSE for FCNs. Convolutional architectures reveal a notable gap: pooling choice matters and finite CNNs can outperform kernels (other work reports ~5–10% gaps). Empirics run up to CIFAR‑10 with channels ≈1000 and ensemble linearization agrees up to a point. infinitely wide = GP NTK frozen kernel

AI-written summary · about

" chapters (19)

1. 0:00Intro
2. 0:45Outline
3. 4:36Starting point
4. 5:53Single-hidden layer (shallow) neural networks of infinite width Consider a NN which
5. 10:21What GP does it correspond to?
6. 11:52Properties of the NNGP
7. 13:19Bayesian inference with a GP prior (Review)
8. 15:05Experiments from original work
9. 16:50Performance comparison
10. 17:45NNGP performance across hyperparameters
11. 18:45Large depth behavior & fixed points
12. 19:56Phase diagrams: experiments vs. theory
13. 20:08Performance trends with width and dataset size
14. 31:21Empirical comparison of various NN-GPS
15. 33:25Empirical trends
16. 34:06Best performing networks: comparison between GPs and SGD-NNS
17. 38:24Partway summary
18. 41:32What dynamics occurs in parameter space?
19. 44:46Closing Remarks

" related talks

• 01h 14mRecent Developments in Over-parametrized Neural Networks, Part IIJason Lee · Deep Learning Boot Camp 2019
• 19mThe Spectrum of Nonlinear Random Matrices for Ultra-Wide Neural NetworksYizhe Zhu · Deep Learning Theory Symposium 2021
• 17mUniversality of Neural NetworksDan Mikulincer · Deep Learning Theory Symposium 2021
• 01h 14mRecent Developments in Over-parametrized Neural Networks, Part IJason Lee · Deep Learning Boot Camp 2019
• 58mIntroduction to the tensor-programs framework, a PL approach that helps analyse ...Hongseok Yang · LAFI 2023
• 01h 20mThe elusive generalization: classical bounds to double descent to grokkingMisha Belkin · Modern Paradigms in Generalization Boot Camp 2024
• 28mNext-generation frameworks for Large-scale Machine LearningAnima Anandkuma · Scale By The Bay 2019
• 34mThe physics of deep learning using tensor networksMarianne Hoogeveen · PyData New York City 2019