Benign, Tempered, or Catastrophic: A Taxonomy of Overfitting
summary
Defines a three-way taxonomy — benign overfitting, tempered overfitting, catastrophic — and claims many interpolating deep nets sit in the middle: nonzero but bounded excess risk. Tools: wide ResNets and MLPs on CIFAR/MNIST/SVHN and synthetic spheres, plus kernel ridge analysis using a Gaussian-design universality heuristic. Key numbers: 30% label flips give clear tempering; models still fully interpolate even at 90% noise. Theory: power-law spectra (e.g. Laplacian kernel or ReLU NTK) yield tempered behavior, ridgeless Gaussian kernel is catastrophic, adding ridge or early stopping restores consistency.
AI-written summary · about
chapters (14)
- 0:00Intro
- 2:16The classical story of overfitting
- 4:55A taxonomy of overfitting
- 6:31A simple experiment
- 13:32Outline
- 13:52(Some) prior works
- 19:28Example methods in the taxonomy
- 25:03Time dynamics of MLP regression
- 29:53Kernel regression can exhibit all three types of fitting
- 37:53The kernel eigensystem
- 41:43KR = linear regression with eigenfunction features
- 43:14Approximation: features are Gaussian and uncorrelated
- 46:52The Trichotomy Theorem
- 52:35Open Questions
related talks
26mIs Overfitting Actually Benign? On the Consistency of Interpolating Methods
48mBenign Overfitting in Linear Prediction
50mFrom Classical Statistics to Modern Machine Learning
01h 14mRecent Developments in Over-parametrized Neural Networks, Part I
01h 14mRecent Developments in Over-parametrized Neural Networks, Part II
06mNeural Networks Demystified: Part 7
46mOn the Foundations of Deep Learning: SGD, Overparametrization, and Generalization
46mKernel and Deep Regimes in Overparameterized Learning