The Spectrum of Nonlinear Random Matrices for Ultra-Wide Neural Networks
summary
Model is a two-layer fully connected network at initialization with iid Gaussian weights and centered, normalized activation; analysis targets the ultra-wide (width >> sample size) regime. Conjugate-kernel and NTK fluctuations, after rescaling by sqrt(width/sample size), converge to a free convolution of a semicircle law with the data spectral law and depend only on the Gaussian expectation of sigma′. Proof uses a nonlinear Hanson–Wright inequality and Hermite expansion. Largest-eigenvalue is tightly concentrated and random-feature vs kernel ridge regression test-error gap vanishes as sample/width → 0.
AI-written summary · about
chapters (10)
- 0:00Intro
- 0:30Fully-connected two-layer neural network
- 1:45Two kernel matrices
- 4:23Semicircle law for sample covariance matrices
- 5:22Assumptions
- 8:14Deformed semicircle law
- 11:26Simulations for Gaussian data
- 12:15Ingredients in the proof
- 13:44Non-asymptotic bound
- 15:37Random feature regression
related talks
01h 14mRecent Developments in Over-parametrized Neural Networks, Part II
49mComputation in Very Wide Neural Networks
42mToward Understanding the Extrapolation of Nonlinear Models to Unseen Domains or the Whole Domain
23mHow Large is the Norm of a Random Matrix?
38mWhy Deep Learning Works: Implicit Self-Regularization in Deep Neural Networks
49mAnalysis of Algorithms on Dense Matrices using Approximate Message Passing
35mSize-free Generalization Bounds for Convolutional Neural Networks
33mThe Power and Limitations of Kernel Learning