A Shallow Introduction to Self-Attention
summary
Starts from perceptrons and RNN failure modes and shows a concrete implementation: embeddings get projected into Q K V projections then scored by scaled dot-product attention (dot products divided by sqrt(dim)) and aggregated with softmax weights. Stacking decoder-only Transformer stacks plus positional embeddings yields autoregressive GPTs; scaling parameters and data produces the reported jump in zero/one/few‑shot performance, and later models add RLHF alignment to tune outputs to human preferences.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 2:06Self-attention popularity
- 7:02Perceptrons to self-attention
- 10:16Refining word embeddings
- 13:50Double-dipping problem QKV
- 16:44Self-attention code
- 18:13Self-attention vs FCNs
- 20:21Transformer block
- 21:25Building GPT
- 22:51GPT1 to modern LMs
- 25:24Final notes and future directions
related talks
01h 10mIntroduction to Transformers
24mRevealing the Unseen: Visualizing Attention in Transformer Models
36mNeural Networks
01h 25mPaper Discussion: Attention is all you need
38mUnderstanding and Applying Self-Attention for NLP
28mEverything you need to know about Transformer Models
01h 29mBuild Your Own Transformer
01h 26mFrom RNNs to Transformers: A Journey through Language Modeling