Special Year on Large Language Models and Transformers, Part 1 Boot Camp
2024-09-04..2024-09-06
Berkeley, United States
who it's for
machine-learning researchers, model training engineers, safety and alignment researchers, applied NLP engineers, research theorists, computer vision researchers
topics
transformer architectures and attention; in-context learning theory; induction heads mechanism; chain-of-thought internalization; scaling laws and data pruning; training pipelines and selection; RLHF and preference optimization; CLIP ablation and segmentation; adversarial suffixes and jailbreaks; signal-propagation trainability theory; modern Hopfield associative memory; ridge versus discrete regression
| title | author | length |
|---|---|---|
| Understanding and Steering Generative AI Systems | Jacob Steinhardt | 01h 04m |
A practical pipeline decomposes CLIP into patches, heads and neurons and shows empirically that mean ablation removing everything but the last ~4 attention… | ||
| Discussion Session | Danqi Chen, Yejin Choi, Daniel Hsu, Aditi Raghunathan | 01h 07m |
A wide-ranging discussion maps specific research bets and failures: controlled data and open‑source training to test combinatorial creativity and diffusion… | ||
| How Do Transformers Work? | Ankur Moitra | 01h 16m |
Explains transformers as an encoder–decoder stack that replaces huge n‑gram parameter blowups (V^4 for a 4‑gram) with learned vector word embeddings (hundreds… | ||
| The Enigma of LLMs: on Creativity, Compositionality, Pluralism, and Paradoxes | Yejin Choi | 01h 18m |
Follow-up experiments claim that LLM behavior surprises: a creativity index matches generated text against large web corpora and finds ChatGPT often reproduces… | ||
| Introduction to Transformers | Daniel Hsu | 01h 10m |
Explains that the core self-attention unit implements an induction heads behavior that requires a two-layer subcircuit (copy key one step, then match query) to… | ||
| Toward Understanding In-context Learning | Tengyu Ma | 01h 30m |
Argues that in‑context learning comes from pretraining on a mixture of HMMs so next‑token Bayes inference implicitly infers a hidden model; proves… | ||
| LLM Safety, Alignment, and Generalization | Roger Grosse | 01h 29m |
Argues that industry needs affirmative safety cases and a science of LLM generalization, not just reassurances about narrowness; describes Anthropic-style… | ||
| On large language models and transformers: perspectives from physics, neuroscience, and theory | Surya Ganguli | 01h 33m |
Scaling laws hold but decay very slowly — test loss drops as a power law with power-law exponent ≈ -0.09, so scaling alone is expensive and unsustainable… | ||
| Training Large Language Models: Practices and Research Questions | Danqi Chen | 01h 38m |
Comprehensive pipeline coverage focuses on concrete claims, tools and numbers: influence-based selection picks 5% of instruction examples and can outperform… | ||
| Challenges in Making LLMs Safe and Robust | Aditi Raghunathan | 01h 27m |
Shows that transformers trained from scratch can implement both ridge vs discrete regression but fine-tuning on one task quickly ruins performance on the… | ||
| nothing matches | ||