The elusive generalization: classical bounds to double descent to grokking
summary
Large neural models and a simple iterative kernel method produce the same surprising pattern: training loss is essentially zero while validation/test accuracy plateaus for many iterations and then rapidly improves — a phenomenon tied not to loss but to feature formation. Empirically an AGOP (average gradient outer product) or the neural feature matrix reveals low‑dim or structured filters; a lightweight procedure called recursive feature machines bootstraps those filters and recovers circulant features for modular arithmetic, explaining sudden out‑of‑domain gains (Fourier‑style multiplication) that neither training nor validation loss predict.
AI-written summary · about
chapters (14)
- 0:00Introduction to generalization
- 1:54Generalization in machine learning
- 8:31Theoretical evolution and regimes
- 12:24Classical analysis and bounds
- 17:31Interpolation and overfitting
- 23:30Double descent and modern trends
- 26:10The phenomenon of grokking
- 30:09Feature learning in neural networks
- 35:48AGOP and feature extraction
- 41:17Recursive feature machines
- 44:45RFM and feature learning
- 47:04Grokking experiments and analysis
- 53:42Modular arithmetic case study
- 58:25Conclusion and future perspectives
related talks
46mOn the Foundations of Deep Learning: SGD, Overparametrization, and Generalization
46mAnalyzing Optimization and Generalization in Deep Learning via Trajectories of Gradient Descent
01h 02mUnexpected Test Losses from Generalization Theory?
50mFrom Classical Statistics to Modern Machine Learning
01h 02mTutorial: Statistical Learning Theory and Neural Networks II
46mKernel and Deep Regimes in Overparameterized Learning
42mWhat (Doesn't) Make Learning Algorithms Generalize?
01h 00mOn continual learning with gradient descent for neural networks