From RNNs to Transformers: A Journey through Language Modeling
summary
Covers tokenization choices (character, word, BPE/subword), shows word embeddings trained via co‑occurrence that produce word2vec analogies, explains recurrent models (RNN/LSTM) with gating and their vanishing‑context limits, then demonstrates how scaled dot‑product multi‑head attention reprojects queries/keys/values to disambiguate senses (apple→phone vs fruit) and enable stacking into Transformer layers. Lists self‑supervised objectives (next‑token, masked, permuted, contrastive) and asks whether tiny LMs can still generate coherent English.
AI-written summary · about
chapters (14)
- 0:00Language modeling journey
- 2:17Language modeling theory
- 3:22Language representation
- 8:03Tokenization methods
- 15:02Subword tokenization
- 22:42Vectorizing tokens
- 29:45Word2Vec and distributed word representations
- 40:13Language modeling challenges
- 45:06Neural language models
- 51:01Recurrent neural networks and LSTMs
- 1:02:56Attention and Transformers
- 1:13:47Self-supervised learning
- 1:19:22Transfer learning
- 1:23:25Summary
related talks
36mNeural Networks
01h 16mHow Do Transformers Work?
49mBen Hall & Barbara Fusinska
25mEmbeddings, Transformers, RLHF: Three key ideas to understand ChatGPT
26mRethinking the Transformer: Toward Native Multimodal Architectures
26mA Shallow Introduction to Self-Attention
01h 00mInside GPT – Large Language Models Demystified
27mEmbeddings, Transfer Learning and More: LLM Tailored to You