Modern paradigms of generalization, the heliocentric model of Aristarchus,...
summary
Four concrete threads: formalize why next-token prediction yields broad downstream performance and why GPT vs BERT differ; explain LLaMA 3’s empirical trick of freeze the validation set then optimize training-data mixtures to hit scaling laws; analyze non‑IID training from overlapping prefixes as a mixing‑time issue (or a replacement concept) for SGD; and make the gradient‑flow tradeoff precise (logsumexp/uniform‑convergence tools) that yields implicit‑bias max‑margin behavior and when those limits actually match practice.
AI-written summary · about
chapters (7)
- 0:00Program introduction
- 2:58Motivations and open problems
- 10:40Generalization challenges
- 19:47Historical scientific anecdotes
- 31:12Scientific method and ML
- 44:02Gradient descent analysis
- 1:02:09Concluding remarks
related talks
10mModern Paradigms in Generalization
42mQuestions for Theory in the New Age of Machine Learning
01h 30mToward Understanding In-context Learning
01h 38mTraining Large Language Models: Practices and Research Questions
43mBeyond the Hype: A Realistic Look at Large Language Models
49mThe Building Blocks of Today’s and Tomorrow’s Language Models
46mPlaying the long game
53mGeneralization insights from actual cognition