Prediction, Generalization, Complexity: Revisiting the Classical View from Statistics Part 2

on YouTube ↗01h 02m

" summary

Shows that classical fixed-X results break down once models approach interpolation and X is random, and gives precise tools: an exact covariance formula for optimism (error minus training error), a Stein identity linking degrees of freedom to the sum of partial derivatives and yielding SURE, and closed forms for linear smoothers (trace of smoother matrix) with the lasso case proving df = number of nonzeros. Contrasts this with double descent behavior when p ≈ n and argues smooth, low-dimensional f keeps the two risks similar.

AI-written summary · about

" chapters (11)

1. 0:00Introduction
2. 1:05Bregman divergences and 0-1 loss
3. 5:09Covariance formula and optimism
4. 12:46Stein's lemma and degrees of freedom
5. 19:03Proof of Stein's lemma
6. 25:57SURE and Stein divergence
7. 32:52Degrees of freedom of lasso
8. 40:52Bootstrap covariance estimates
9. 43:54Random vs fixed x risk
10. 50:37Empirical process theory insights
11. 1:01:00Summary

" related talks

• 50mFrom Classical Statistics to Modern Machine LearningMikhail Belkin · Frontiers of Deep Learning 2019
• 01h 22mGeneralization IVPeter Bartlett, Sasha Rakhlin · Deep Learning Boot Camp 2019
• 42mWhat (Doesn't) Make Learning Algorithms Generalize?Roi Livni · Lower Bounds, Learning, and Average-Case Complexity
• 48mBenign Overfitting in Linear PredictionPeter Bartlett · Frontiers of Deep Learning 2019
• 01h 01mRobustness, Stochastics, Uncertainty 1Sasha Rakhlin, Ben Recht, Laurent El Ghaoui · Bridging Continuous and Discrete Optimization Boot Camp 2017
• 56mSparsifying Generalized Linear ModelsYang Liu · Optimization and Algorithm Design 2023
• 53mDesigning Robust LearnersJacob Steinhardt · Emerging Challenges in Deep Learning (Simons Institute workshop, 2019)
• 01h 01mRobustness, Stochastics, Uncertainty 2Sasha Rakhlin, Ben Recht, Laurent El Ghaoui · Bridging Continuous and Discrete Optimization Boot Camp 2017