Feature Selection with Gradient Descent on Two-layer Networks in Low-rotation Regimes

on YouTube ↗58m

" summary

Gradient flow and SGD on two‑layer ReLU nets show concrete selection and generalization claims in low‑rotation regimes: near initialization they exit the NTK after O(sqrt(m)) weight motion and require width and sample count ~1/NTK‑margin, with GF provably reaching a constant NTK margin rather than just nondecreasing margins; in Neural Collapse data sample complexity scales with the number of groups; if inner weights are forbidden to rotate GF at large width attains global max margin (sample complexity ~1/margin) using new potential functions and exploiting a root‑M Frobenius norm movement, though some proofs need exponential width in d.

AI-written summary · about

" chapters (19)

1. 0:00Introduction
2. 1:50Margin as a technical tool
3. 3:42Global max margin results
4. 6:24Technical contributions
5. 8:42Sparse parity experiments
6. 12:11Analyzing network width
7. 14:24Sample complexity results
8. 17:30Global margin maximization
9. 20:23SGD theorem breakdown
10. 24:30Proof mechanics
11. 29:43Gradient flow theorem
12. 32:20Generalization bounds
13. 36:16KKT points and simplicity
14. 38:30Low rotation regime
15. 42:19Global max margin for scalars
16. 44:00Potential function derivation
17. 48:29Weight ratios and monsters
18. 53:03Summary and future work
19. 55:23Q&A session

" related talks

• 46mAnalyzing Optimization and Generalization in Deep Learning via Trajectories of Gradient DescentNadav Cohen · Frontiers of Deep Learning 2019
• 01h 01mOptimization's Untold Gift to Learning: Implicit RegularizationNathan Srebro Bartom · Optimization, Statistics and Uncertainty
• 46mOn the Foundations of Deep Learning: SGD, Overparametrization, and GeneralizationJason Lee · Frontiers of Deep Learning 2019
• 11mOn Implicit Regularization in Deep LearningWei Hu · Meet the Fellows Welcome Event
• 01h 20mThe elusive generalization: classical bounds to double descent to grokkingMisha Belkin · Modern Paradigms in Generalization Boot Camp 2024
• 50mLearning and Generalization in Over-parametrized Neural Networks, Going Beyond KernelsYuanzhi Li · Frontiers of Deep Learning 2019
• 45mIs Deeper Better Only When Shallow Is Good?Shai Shalev-Shwartz · Emerging Challenges in Deep Learning (Simons Institute workshop, 2019)
• 42mA Primal-dual Analysis of Margin Maximization by Steepest Descent MethodsMatus Telgarsky · Frontiers of Deep Learning 2019