Keys, Queries, and Values: The Celestial Mechanics of Attention
summary
Claims that attention is what makes transformers understand context, demonstrates the actual scaled dot‑product formula softmax of QK^T/√dk and shows embeddings (dozens to thousands of dimensions) move into context by replacing a word vector with a weighted mixture of others (weights = e^{score}/sum). Uses dot product and cosine intuition, then implements asymmetry by applying two learned linear transforms so keys pull, queries get pulled. Presents concrete numbers (divide by √dk, exponentiate, normalize to percentages) and visualizes the result working on ambiguous words.
AI-written summary · about
chapters (7)
- 0:00Attention mechanisms intro
- 4:19What embeddings are
- 11:12Attention modifies embeddings
- 22:10Key/query matrices for similarity
- 33:37Asymmetric pulls with keys/queries
- 39:41Keys/queries in practice
- 43:54Summary
related talks
24mRevealing the Unseen: Visualizing Attention in Transformer Models
01h 10mIntroduction to Transformers
26mA Shallow Introduction to Self-Attention
48mThe Magic Of attention in LLMS
41mLong-context Attention in Near-Linear Time
34mAttention is not all you need - Keynote
01h 16mHow Do Transformers Work?
01h 25mPaper Discussion: Attention is all you need