Reconsidering Overfitting in the Age of Overparameterized Models
summary
Empirical and theoretical claims: overparameterized linear models show double descent because variance decays as dimension grows while bias rises; for isotropic Gaussians the min-L2 interpolator yields variance ≈ n/(d−n−1) and bias ≈ (d−n)/d so no consistency, but spiked covariances reduce bias. Methods: Gaussian design, inverse-Wishart/rand matrix calculations, convex Gaussian minimax and localized uniform convergence. New statistical claim: min-L1 norm interpolation gives O(1/log n) error, while intermediate Lp interpolators (1<p<2) can achieve polynomial decay approaching 1/n with enough overparameterization.
AI-written summary · about
chapters (10)
- 0:00Classical vs. modern theory
- 5:03Empirical double descent
- 10:46Motivations for theory
- 13:22Analyzing linear interpolators
- 18:42Variance and bias dynamics
- 36:24Impact of data distribution
- 40:00Bias in overparameterization
- 45:33Covariance spikiness impact
- 46:03Structure of interpolators
- 50:45Inductive bias and p-norms
related talks
01h 04mUniform Convergence of Interpolators: Gaussian Width, Norm Bounds, and Benign Overfitting
48mBenign Overfitting in Linear Prediction
29mImplicit Regularization in Nonconvex Statistical Estimation
26mIs Overfitting Actually Benign? On the Consistency of Interpolating Methods
55mThe Devil is in the Tails and Other Stories of Interpolation
48mComputational-Statistical Tradeoffs in Robust High-Dimensional Estimation
01h 22mGeneralization IV
53mDesigning Robust Learners