Overview of Statistical Learning Theory Part 2
summary
Surveys classical tools and then ties them to optimization: uniform convergence via fat-shattering dimension and Rademacher complexity gives sample bounds (scale-sensitive, norms and margins → roughly 1/γ^2), Lipschitz losses let one pass from predictors to loss classes, and optimization methods produce implicit bias (GD → min‑L2/max‑margin, coordinate descent a different bias). Shows regularization and stability as an alternate route, gives a simple counterexample (center‑of‑mass with missing data) where ERM fails despite learnability, and stresses worst‑case limits versus distribution‑dependent phenomena.
AI-written summary · about
chapters (8)
- 0:00Continuous complexity measures
- 1:04Discrete to continuous classes
- 10:21Covering numbers and dimensions
- 21:55Norm and margin in learning
- 36:38Optimization algorithms and bias
- 43:31Uniform convergence and learnability
- 57:01Stability versus uniform convergence
- 1:00:13Limitations of worst-case analysis
related talks
01h 05mOptimization for Machine Learning I
59mTutorial: Statistical Learning Theory and Neural Networks I
01h 01mRobustness, Stochastics, Uncertainty 1
01h 01mRobustness, Stochastics, Uncertainty 2
01h 01mOptimization's Untold Gift to Learning: Implicit Regularization
01h 23mImplicit Regularization II
42mWhat (Doesn't) Make Learning Algorithms Generalize?
50mFrom Classical Statistics to Modern Machine Learning