How neural networks learns simple functions?
summary
Examines learning of multi‑index targets on Gaussian data and shows concrete failure modes and fixes: random features = polynomial fitting so kernels or lazy nets only recover degree up to min(n,p) and need n,p ~ d^k to hit kth Hermite term; AMP (approximate message passing) achieves optimal recovery with n ~ d but is impractical; surprisingly, a single large gradient step or two‑step SGD with repeated data breaks Hermite barrier and lets SGD learn higher‑order components with n scaling like d or d log d; constructs a three‑layer target with r~√d projections and demonstrates a provable depth advantage (empirically confirmed, partially proved).
AI-written summary · about
chapters (15)
- 0:00Workshop introduction
- 5:27Speaker: Florent Krzakala
- 6:53Simple functions in ML
- 8:19Multi-index functions
- 11:55Random features and kernels fail
- 13:43Gradient step transforms learning
- 16:31Easy vs hard functions
- 23:42Connecting physics, statistics, and computer science
- 26:54Neural networks learn directions
- 31:14Information exponent limits
- 34:10Training repetition helps
- 38:30Two-step gradient descent
- 41:28Depth in 3-layer networks
- 47:21Summary and open questions
- 49:10Q&A
related talks
50mLearning and Generalization in Over-parametrized Neural Networks, Going Beyond Kernels
01h 04mLearning Staircases
01h 51mUnderstanding Neural Networks (Somewhat)
32mRobust Learning of a Single Neuron via Sharpness
01h 20mThe elusive generalization: classical bounds to double descent to grokking
32mLearning One-hidden-layer Neural Networks with Landscape Design
20mFeature Learning: From Optimization to Modern Generalization
01h 30mAnalytic Methods for Supervised Learning IV