Does generalization imply accuracy on the line? A new look at robust generalization
summary
Claim: apparent “emergent” jumps often come from metric choice rather than a new model ability. Reanalysis of BIG-Bench plus arithmetic experiments and a simple probabilistic toy model show 35 of 39 metrics do not display emergence; the sharp jumps appear for discontinuous, exact-match or argmax-based scores and disappear with token-add distance or Brier score (partial-credit, continuous metrics). Experiments use survival-function correlations and Spearman fits, and demonstrate that normalizing to multiple-choice options and then taking accuracy degrades predictability.
AI-written summary · about
chapters (8)
- 0:00Robust generalization intro
- 1:00Scaling laws in model performance
- 10:00Metrics and downstream unpredictability
- 25:30Emergent abilities from metric choice
- 40:00Evaluating multiple-choice benchmarks
- 53:40Follow-up on multiple-choice evaluation
- 1:02:45Metric choice for predictability
- 1:04:54Q&A on metric types
related talks
01h 10mModern paradigms of generalization, the heliocentric model of Aristarchus,...
42mWhat (Doesn't) Make Learning Algorithms Generalize?
01h 20mThe elusive generalization: classical bounds to double descent to grokking
53mGeneralization insights from actual cognition
36mThe Limits of Post-Selection Generalization
01h 01mRobustness, Stochastics, Uncertainty 1
34mDistributional Robustness, Learning, and Empirical Likelihood
01h 02mPrediction, Generalization, Complexity: Revisiting the Classical View from Statistics Part 2