Introduction to Transformers
Talkat Special Year on Large Language Models and Transformers, Part 1 Boot Camp · 2024-09-04..09-06·
summary
Explains that the core self-attention unit implements an induction heads behavior that requires a two-layer subcircuit (copy key one step, then match query) to retrieve the last-next token; composing k simple functions requires only logarithmic depth (O(log k)) rather than linear depth. Describes autoregressive training via a log‑linear head, chain‑of‑thought giving extra intermediate tokens that aid learning, and connects self‑attention to modern Hopfield associative memory whose update rule matches attention and can store exponentially many patterns.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 0:14Transformer architecture
- 1:47Self-attention mechanism
- 6:06Attention patterns
- 11:23Induction heads
- 14:54Implementing induction
- 21:00In-context learning
- 30:11Function composition
- 37:46Theoretical perspectives
- 42:44Learning theory
- 54:30Chain of thought
- 58:13Prediction vs generation
- 1:03:42Associative memories
related talks
26mA Shallow Introduction to Self-Attention
24mRevealing the Unseen: Visualizing Attention in Transformer Models
38mUnderstanding and Applying Self-Attention for NLP
38mHow Transformers Learn Causal Structure with Gradient Descent
01h 25mPaper Discussion: Attention is all you need
26mRethinking the Transformer: Toward Native Multimodal Architectures
01h 26mFrom RNNs to Transformers: A Journey through Language Modeling
40mAn Occam's Razor Principle for Transformers?