Toward Understanding In-context Learning
Talkat Special Year on Large Language Models and Transformers, Part 1 Boot Camp · 2024-09-04..09-06·
summary
Argues that in‑context learning comes from pretraining on a mixture of HMMs so next‑token Bayes inference implicitly infers a hidden model; proves linear‑regression prompts reduce to Bayesian/ridge estimators and that a shallow linear‑attention transformer implements one‑step gradient descent while deeper nets can approximate ridge regression; shows larger models (text‑davinci‑002 / GPT variants) actually learn flipped labels whereas RLHF‑tuned models sometimes resist, and flags limits: population‑loss assumptions, expressivity and sample‑complexity gaps.
AI-written summary · about
chapters (10)
- 0:00In-context learning intro
- 1:15What is in-context learning
- 8:44Two learning types in-context
- 11:35Theoretical explanations
- 29:07Hidden Markov models
- 39:15Linear regression model
- 57:57Empirical validation
- 1:01:49Model size impact
- 1:03:13Open questions on reasoning
- 1:29:24Q&A
related talks
46mLearning Theory of Transformers: Generalization and Optimization of In-Context Learning
25mEmbeddings, Transformers, RLHF: Three key ideas to understand ChatGPT
01h 10mModern paradigms of generalization, the heliocentric model of Aristarchus,...
43mThe Brain’s Guide to Dealing with Context in Language Understanding
39mComputational Thinking on Learning Models
01h 00mThe Frontier between Retrieval-augmented and Long-context Language Models
11mCollaborative Natural Language Inference
41mSub-Linear Algorithms meets Large Language Models