Training on the Test Set and Other Heresies
summary
Claims that huge models always overfit get tested by reproducing CIFAR-10 and ImageNet test sets: researchers resample Tiny Images and rerun Mechanical Turk labeling, evaluate archived GitHub weights (VGG, ShakeShake, random-feature baselines, Inception, models up to 600 million parameters) and find substantial accuracy drops — roughly 8–12% drop on CIFAR-10 and about 10% drop on ImageNet — yet better models still beat worse ones, giving no adaptive overfitting signal and pointing to fragile performance under human-label distribution shift.
AI-written summary · about
chapters (22)
- 0:00Intro
- 0:45Conventional wisdom
- 2:13What is machine learning
- 3:04What is generalization
- 5:09What can we take away
- 6:51Least favorite figure
- 8:57Inception model
- 10:19Regularization
- 12:04Pull Request
- 12:59Random Features
- 13:20Regression
- 14:36Boosting
- 15:14Model Size
- 17:38Diminishing Returns
- 18:38New Holdout Set
- 20:11New Test Set
- 22:05Results
- 31:21Mechanical Turk
- 34:04Variability
- 43:37Imagenet Data
- 44:29Cotton Pickers
- 46:08Kaggle
related talks
01h 05mCan you trust your (large language) model?
33mBuild Better AI with First-Party Data and Fine-Tuning
34mNeural networks: insane in the membrane
37mOptimal on Paper, Broken in Reality
30mActive Learning and Human-in-the-Loop Machine Learning
28m🧼 From GPU-poor to data-rich: data quality practices for LLM fine-tuning
28mCifar-10 Exploratory Data Analysis
01h 04mCan you trust your (large language) model?