Learning Theory of Transformers: Generalization and Optimization of In-Context Learning

on YouTube ↗46m

" summary

Claims that pretrained Transformers reach minimax optimality for in‑context regression by learning shared nonlinear features and then doing linear (ridge) readout; analysis uses a truncated infinite‑feature model with context length n and T pretraining tasks and proves test risk converges to Bayes optimal as T grows. Optimization guarantees come from mean‑field two‑layer networks, Wasserstein gradient flow and a strict‑saddle property that lets gradient dynamics escape saddles; experiments show learned linear attention approximates ridge regression and matches the theory.

AI-written summary · about

" chapters (5)

1. 0:00Transformer learning theory intro
2. 4:44In-context learning statistics
3. 16:32Linear attention & ridge regression
4. 31:40Mean field optimization
5. 42:44Summary & open challenges

" related talks

• 01h 30mToward Understanding In-context LearningTengyu Ma · Special Year on Large Language Models and Transformers, Part 1 Boot Camp
• 01h 27mChallenges in Making LLMs Safe and RobustAditi Raghunathan · Special Year on Large Language Models and Transformers, Part 1 Boot Camp
• 01h 33mOn large language models and transformers: perspectives from physics, neuroscience, and theorySurya Ganguli · Special Year on Large Language Models and Transformers, Part 1 Boot Camp
• 40mAn Occam's Razor Principle for Transformers?John Langford · The Role of TCS in Modern Machine Learning
• 01h 01mOptimization's Untold Gift to Learning: Implicit RegularizationNathan Srebro Bartom · Optimization, Statistics and Uncertainty
• 38mHow Transformers Learn Causal Structure with Gradient DescentJason Lee · Domain Adaptation and Related Areas 2024
• 01h 23mImplicit Regularization IINati Srebro · Deep Learning Boot Camp 2019
• 01h 16mHow Do Transformers Work?Ankur Moitra · Special Year on Large Language Models and Transformers, Part 1 Boot Camp