Tutorial: Statistical Learning Theory and Neural Networks I
summary
Review of uniform laws of large numbers and concrete complexity tools: VC dimension (upper bound ≈ O(P·L) for ReLU nets) and Rademacher complexity, with a simple bound scaling like 2·B^L / sqrt(n) when each layer norm ≤ B. Empirical risk minimization (ERM) is analyzed via these measures; large parameter counts P alone don’t force generalization failure but large layerwise norms do. Demonstrated phenomenon: with 15% label noise gradient descent finds interpolators (100% train, ~85% test), exposing limits of uniform-convergence guarantees.
AI-written summary · about
chapters (11)
- 0:00Welcome and logistics
- 4:20Supervised learning setup
- 5:57Risk decomposition
- 10:34Empirical risk minimization
- 16:02Uniform convergence
- 18:22Complexity and generalization
- 22:54VC dimension
- 31:19Rademacher complexity
- 35:08Rademacher calculus
- 38:20Neural network theory
- 54:41Concluding remarks
related talks
01h 03mOverview of Statistical Learning Theory Part 2
50mLearning and Generalization in Over-parametrized Neural Networks, Going Beyond Kernels
01h 04mRandomized Embeddings and Neural Networks
28mTowards the Universal Law of Robustness for Deep Neural Networks
01h 23mGeneralization II
01h 04mNeural Networks meet Nonparametric Regression: Generalization by Weight Decay and Large...
17mUniversality of Neural Networks
13mThe dynamical view of machine learning: implicit biases of large learning rates