How Do Transformers Work?
Talkat Special Year on Large Language Models and Transformers, Part 1 Boot Camp · 2024-09-04..09-06·
summary
Explains transformers as an encoder–decoder stack that replaces huge n‑gram parameter blowups (V^4 for a 4‑gram) with learned vector word embeddings (hundreds to thousands of dims), inner‑product attention matrix (softmax over query/key dot products) and multi‑head attention, plus positional encodings and causal masking. Shows tools like SVD/word2vec for embeddings, layerwise feedforward nets, and six‑layer originals; notes transformers are highly parallelizable, recover long‑range dependencies, and empirical work (e.g., reverse‑engineering GPT2‑small) finds concrete circuits that copy/backup and resolve pronouns.
AI-written summary · about
chapters (8)
- 0:00Welcome
- 7:37Boot camp overview
- 9:47Language model history
- 23:14Transformer architecture
- 34:04Attention mechanisms
- 55:46Positional embeddings
- 1:06:25Transformer computations
- 1:10:01Q&A
related talks
38mFrom Von Neumann to Transformers: An Infra Engineer's Guide to What's Actually Inside an LLM.
20mMechanistic Interpretability
01h 26mFrom RNNs to Transformers: A Journey through Language Modeling
34mBeyond Transformers - Intro to RWKV Architecture & The World To...
17mHow Transformers Finally Ate Vision
26mRethinking the Transformer: Toward Native Multimodal Architectures
24mRevealing the Unseen: Visualizing Attention in Transformer Models
34mTransformers End to End Experience training...