On large language models and transformers: perspectives from physics, neuroscience, and theory
summary
Scaling laws hold but decay very slowly — test loss drops as a power law with power-law exponent ≈ -0.09, so scaling alone is expensive and unsustainable. Analytic toy models (random feature and student–teacher setups) tie those exponents to data kernel spectra and show multi-descent from multiscale data, removable by optimal regularization. Data pruning (keep hard examples after short probes) provably beats power laws for perceptrons and halves LAION training cost in practice, giving pruning yields exponential error decay. Diffusion analytic ODEs explain U‑Net samples as a local patch jigsaw of training patches. Signal‑propagation theory predicts transformer trainability at the intersection of forward chaos and backward gradient stability, and in‑context learning shows a task-diversity threshold ∝ dimension that unlocks exponentially many tasks.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 0:42Theory of deep learning
- 5:22Scaling laws and limits
- 9:14Power law spectra and redundancy
- 15:00Pruning datasets to beat power laws
- 22:29Data pruning applications
- 27:52Toy models of transformers
- 39:16Exponential creativity
- 43:57Diffusion models and creativity
- 51:38In-context learning
- 1:05:30Transformer performance prediction
- 1:16:29Neuroscience and brain modeling
- 1:23:12Summary and future
related talks
36mScaling Laws, Emergent Behaviors, and AI Democratization
26mThe scaling laws of AI Why neural networks continue to grow
27mLarge Language Models and the Future of AI
01h 04mTransformers can learn compositional function
46mLearning Theory of Transformers: Generalization and Optimization of In-Context Learning
40mIs my Large Language Model a Strange Loop?
40mAn Occam's Razor Principle for Transformers?
01h 01mBetween the Layers– Interpreting Large Language Models