The Devil is in the Tails and Other Stories of Interpolation
summary
Shows that reweighting fails for exponentially-tailed losses but recovers when using a polynomially-tailed loss, with theory and CIFAR/CelebA runs verifying improved worst-group accuracy. For a label-shift Gaussian model the theory prescribes overcorrection (best empirical scaling w = τ^3), yielding test error that decays exponentially in minority sample count times ||μ||^4/d, while reweighted cross-entropy stays constant. Second result gives a worst-case: the minimum L1-norm interpolant can have excess risk that shrinks exponentially slower than the L2 interpolant, e.g. L2 ~1/n vs L1 ~1/log n in a constructed high‑dimensional regime.
AI-written summary · about
chapters (26)
- 0:00Intro
- 1:47Interpolating Training Data can be beneficial
- 2:53What happens when veer off this standard path?
- 4:19Is Interpolating at odds with Robustness?
- 6:15Vignette II: Training Sparse Models
- 6:50Example: Large Language Models
- 8:58Classification under Distribution Shift Consider a binary classification task with distribution shift
- 9:33Distribution Shift and Importance Weighting
- 10:16Doesn't work
- 11:14Why do exp-tailed losses fail?
- 12:26Intuition: Reweighting doesn't affect Exp-Tailed Losses
- 13:36Our Proposal: Switch losses log 1 + exp
- 14:25Implicit bias for poly-tailed losses
- 20:49But what about the test performance?
- 21:21What about the Test Error? Want to study the generalization error in the overparameterized regime with distribution shift
- 26:04Exponentiate the weights and train on biased loss
- 30:33Loss Ratio Bound
- 31:59Experiments with Neural Network Classifiers
- 34:02Comparison with Stronger Reweighting Methods
- 34:38Takeaways
- 37:58Is sparsity incompatible with interpolation?
- 38:09Sparsity in Linear Regression
- 40:11Construction for the Lower bound
- 41:58Lower bound for Any Sparse Linear Interpolator
- 43:51What about the Min 6-norm Interpolant?
- 47:56Summary and Future Directions
related talks
01h 18mReconsidering Overfitting in the Age of Overparameterized Models
26mIs Overfitting Actually Benign? On the Consistency of Interpolating Methods
01h 04mUniform Convergence of Interpolators: Gaussian Width, Norm Bounds, and Benign Overfitting
50mFrom Classical Statistics to Modern Machine Learning
48mBenign Overfitting in Linear Prediction
41mTransfer learning via local convergence rates of the nonparametric least squares estimator
53mDesigning Robust Learners
42mToward Understanding the Extrapolation of Nonlinear Models to Unseen Domains or the Whole Domain