Unknown Futures of Generalization 2024
2024-12-02..2024-12-06
-- online --
[Emerging Generalization Settings 2024, Understanding Higher-Level Intelligence from AI, Psychology, and Neuroscience Perspectives, The Future of Language Models and Transformers 2025, Modern Paradigms in Generalization Boot Camp 2024, Frontiers of Deep Learning 2019, The Role of TCS in Modern Machine Learning]
who it's for
language model researchers, machine learning theorists, cognitive neuroscientists, computational linguists, protein engineers, AI ethics researchers
topics
in-context learning theory; algorithmic stability via bagging; transformer minimax optimality; memorization in logical reasoning; residual stream feature decoding; temporal context and traveling waves; first-person fairness in chatbots; protein structure to sequence design; generalization via analogy; learning under distribution shift; low-data small-brain generalization; post-training proof verification with Lean
| title | author | length |
|---|---|---|
| Debate: Sparks versus embers | Sebastien Bubeck, Tom McCoy, Pavel Izmailov, Ankur Moitra, Anil Ananthaswamy | 01h 22m |
Motion: whether current LLM scaling plus realistic post-training can produce new proof techniques to settle problems like P≠NP. Panelists present concrete… | ||
| Off-the-shelf Algorithmic Stability | Rebecca Willett | 54m |
Bagging is shown to stabilize any black‑box learner by averaging models trained on random bags and yields provable epsilon‑delta stability where delta… | ||
| Learning Theory of Transformers: Generalization and Optimization of In-Context Learning | Taiji Suzuki | 46m |
Claims that pretrained Transformers reach minimax optimality for in‑context regression by learning shared nonlinear features and then doing linear (ridge)… | ||
| First-Person Fairness in Chatbots | Adam Tauman Kalai | 48m |
Defines first-person fairness as fairness toward the individual user and gives a repeatable pipeline: use public chat corpora (LMSYS, WildChat), a… | ||
| Panel on the future of scientific research and education | Konrad Kording, Florent Krzakala, Shirley Ho, Joshua Batson, Adam R. Klivans, Fanny Yang, Russell Impagliazzo | 01h 08m |
A live panel debates which research skills survive widespread LLM use and gives concrete examples: a student-run “Resurrected Scientists” ChatGPT experiment… | ||
| Generalization in the representations and computations of frontier language models. | Joshua Batson | 52m |
Walks through geometry of LLM activations and shows a concrete decoding method using sparse autoencoders applied to the residual stream of Claude 3 Sonnet… | ||
| Temporal Context in Brains and AI | Terry Sejnowski | 54m |
Argues that sequences and hours-long working memory arise from cortical traveling waves coupled to spike-timing plasticity: gamma packets (~100 ms) produce… | ||
| Frame-shifting and Conceptual Blending: What do large language models have to say? | Seana Coulson | 59m |
Argues that EEG/ERP data (N400, P600) from 160 joke stimuli (subjects see 80 jokes, 80 straight endings) align with language-model surprisal measures: GPT-3… | ||
| How machine learning is influencing protein engineering | Jennifer Listgarten | 53m |
Field reports that AlphaFold and inverse folding models enable structure-to-sequence pipelines but fail for enzymes and dynamic conformations; the work shows… | ||
| Strong Generalization from Small Brains and No Training Data | Bruno Olshausen | 58m |
Argues small brains form low-power geometric models and demonstrates concrete algorithms: a resonator network that couples two Hopfield-style attractors to… | ||
| How neural networks learns simple functions? | Florent Krzakala | 56m |
Examines learning of multi‑index targets on Gaussian data and shows concrete failure modes and fixes: random features = polynomial fitting so kernels or lazy… | ||
| The Curious Incident of Developing Artificial General Intelligence | Jimmy Ba | 45m |
Argues that AGI progress mainly comes from engineering and compute, not new theory, and that xAI builds and runs a 200K GPUs cluster with 50× faster chips but… | ||
| Fireside Chat | Peter Bickel, Po-Ling Loh, Haiyan Huang | 50m |
Career arc and concrete contributions cover a decades-long mix of theory and applied work: a Science paper that resolves Berkeley grad admissions / Simpson's… | ||
| On Memorization of Large Language Models in Logical Reasoning | Chiyuan Zhang | 48m |
Paper measures memorization on a procedurally generated Knights-and-Knaves benchmark and defines a per-sample memorization score that multiplies accuracy with… | ||
| Convex Analysis at Infinity: An Introduction to Astral Space and Fireside Chat | Robert Schapire | 54m |
Constructs a compact extension of R^n called astral space that adds directional limits so every linear map extends continuously, proves any convex f on R^n has… | ||
| Fireside Chat | Robert Schapire | 21m |
Explains why boosting drives weights toward infinity yet still generalizes: PAC-style proofs predict overfitting but experiments do not, and margin theory… | ||
| A New Paradigm for Learning with Distribution Shift | Adam R. Klivans | 48m |
A new model called TDS learning requires an efficient test on training data that may reject or else outputs a classifier with provable low test error even… | ||
| Understanding the abilities of AI systems: Memorization, generalization, and points in between | Tom McCoy | 44m |
Argues that LLM abilities are best diagnosed by comparing outputs to training data and shows concrete tests and numbers: generated text has 93% novel 10‑grams… | ||
| Weak-to-Strong Generalization | Pavel Izmailov | 43m |
Paper demonstrates that fine-tuning larger GPT-4–family models on labels produced by smaller models often yields better performance than those weak supervisors… | ||
| Generalization insights from actual cognition | Konrad Kording | 53m |
Opens by making malicious fun of out-of-distribution and argues generalization is about structure that maps past to future, not tiny distribution shifts. Uses… | ||
| Generalization via analogy in young children and Large Models. | Alison Gopnik | 48m |
Empirical tests show toddlers use relational rules to generalize: 24‑month‑olds (n=100 preregistered) pick the right machine to make a hat fit after seeing a… | ||
| The Wisdom of the Body Revisited | Ben Recht | 49m |
Argues that many non‑neural networks implement proportional–integral control, modeling blood calcium regulation around ~9 mg/dL as a PI loop and proving that… | ||
| nothing matches | ||