Generalization II

on YouTube ↗01h 23m

" summary

Shows that uniform deviation between population and empirical losses is essentially captured by Rademacher averages using symmetrization with a ghost sample and ±1 random signs, with tight upper and lower bounds up to constants. Tools: contraction (Lipschitz losses), covering numbers/Dudley integral, and layerwise Frobenius or 1‑norm scaling. Consequence: binary losses drop to Rademacher of predictors, sigmoid nets give O(B^2 sqrt(d/n)), ReLU nets scale like product‑of‑layer‑norms / sqrt(n).

AI-written summary · about

" chapters (9)

1. 0:00Recap
2. 2:51Abstract
3. 12:45Random Averages
4. 17:01Examples
5. 27:11The Big Theorem
6. 36:51Losses
7. 43:21Deep Networks
8. 45:01Sigmoid
9. 58:11Maximum Over Functions

" related talks

• 59mTutorial: Statistical Learning Theory and Neural Networks ISpencer Frei · Deep Learning Theory Workshop and Summer School 2022
• 01h 01mRobustness, Stochastics, Uncertainty 2Sasha Rakhlin, Ben Recht, Laurent El Ghaoui · Bridging Continuous and Discrete Optimization Boot Camp 2017
• 29mOn Learning Distributions from their SamplesSudeep Kamath · Information Theory, Learning and Big Data (Simons Institute, 2015)
• 01h 03mOverview of Statistical Learning Theory Part 2Nati Srebro · Modern Paradigms in Generalization Boot Camp 2024
• 53mDesigning Robust LearnersJacob Steinhardt · Emerging Challenges in Deep Learning (Simons Institute workshop, 2019)
• 01h 01mRobustness, Stochastics, Uncertainty 1Sasha Rakhlin, Ben Recht, Laurent El Ghaoui · Bridging Continuous and Discrete Optimization Boot Camp 2017
• 01h 02mUnexpected Test Losses from Generalization Theory?Frederic Koehler · Emerging Generalization Settings 2024
• 01h 30mHigh-Dimensional Statistics IIMartin Wainwright · Big Data Boot Camp 2013