Towards the Universal Law of Robustness for Deep Neural Networks
summary
Claims that overparametrization is necessary for robustness, showing memorization requires roughly 2n/d neurons and that smooth (Lipschitz-bounded) networks need parameter count D×K far larger than n. Uses high-dimensional geometry and topology with pictorial constructions to illustrate the BLN Conjecture (2020) and extreme cases from the Bubeck-Sellke Theorem (2021), and points to partial proofs developed in industry (Microsoft). Demonstrations argue Lipschitz constants decrease as neuron count K increases, so bigger nets yield provable smoothness.
AI-written summary · about
chapters (8)
- 0:00Neural network robustness intro
- 1:12How networks classify data
- 5:55ReLU in space folding
- 7:46Approximation vs memorization
- 9:31Curse of dimensionality
- 13:34Over-parametrization ensures robustness
- 17:28Robustness theorem corollaries
- 22:49Q&A
related talks
59mTutorial: Statistical Learning Theory and Neural Networks I
46mOn the Foundations of Deep Learning: SGD, Overparametrization, and Generalization
01h 14mRecent Developments in Over-parametrized Neural Networks, Part I
01h 14mRecent Developments in Over-parametrized Neural Networks, Part II
50mLearning and Generalization in Over-parametrized Neural Networks, Going Beyond Kernels
01h 51mUnderstanding Neural Networks (Somewhat)
01h 04mNeural Networks meet Nonparametric Regression: Generalization by Weight Decay and Large...
01h 21mTutorial: Implicit Bias I