Transformers Inside Out (Parts 1 & 2)
summary
A runnable mini‑GPT gets built with nanoGPT and PyTorch, trained on a ~35,000 knock‑knock corpus (max sequence 15, vocab ≈10,000) using a custom prepare.py and debugger‑driven walkthrough. Starting with tiny models fails; moving to a distilled GPT‑2 style stack then squeezing to one attention head and 36 embedding dimensions yields syntactically valid knock‑knock outputs (they work as knock‑knock jokes but are not reliably funny), plus interactive visualisations of embeddings, Q/K/V flows and token logits.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 3:12What is a transformer
- 8:32Visualizing embeddings
- 18:19Setting up NanoGPT
- 27:12Positional encoding
- 36:28Attention mechanisms
- 58:20Transformer block structure
- 1:08:55Feed forward networks
- 1:17:27Generating jokes
- 1:21:12Q&A
related talks
28mGPT or how I learned to stop worrying and love the generative AI
01h 00mInside GPT – Large Language Models Demystified
45mTransformers for Game Log Data
47mInside GPT – Large Language Models Demystified
01h 01mInside GPT – Large Language Models Demystified
01h 01mKeynote: Transformers: The Rise of ChatGPT
38mFrom Von Neumann to Transformers: An Infra Engineer's Guide to What's Actually Inside an LLM.
45mHow to use GPT in your projects: A Case Study