The Building Blocks of Today’s and Tomorrow’s Language Models
summary
Survey of 2025 LLM architectures highlights concrete tradeoffs and wins: models like DeepSeek v3 reach 671 billion parameters while using mixture of experts (256 experts, ~37B active per token) to cut inference cost; grouped query attention and multi‑head latent attention both shrink KV cache with MLA giving better modeling at similar compression; sliding window attention limits lookback and is combined with GQA in Gemma 3; hierarchical reasoning models (tiny reasoning model) score strongly on ARC, and text diffusion trades serial chains of thought for parallel generation.
AI-written summary · about
chapters (22)
- 0:00Introduction
- 1:05LLM landscape 2025
- 2:10LLM architecture trends
- 3:53Grouped Query Attention
- 7:03Multi Head Latent Attention
- 10:18Sliding Window Attention
- 13:51Mixture of Experts
- 17:25Beyond transformers
- 20:17Hierarchical reasoning models
- 22:45Text diffusion models
- 24:08Emerging architectures
- 27:36Training vs inference vs reasoning
- 31:11Inference challenges
- 34:21Fine-tuning MoE models
- 35:47Architecture decisions
- 38:53Attention robustness
- 40:43Motivation for attention variants
- 41:05Discrete text diffusion
- 43:11Chip design considerations
- 45:59Explainability challenges in large language models
- 47:24Future interpretability
- 49:08Q&A
related talks
27mThe Cutting Edge of AI: From LLMs to Multimodal Intelligence
24mHarnessing the Power of LLMs in Advanced Analytics
36mInsights from The State of AI Report 2023
29mThe Evolution of LLMs: From Imitation to Reasoning
33mDeep research at scale
40mThe Future of Language Models
43mBeyond the Hype: A Realistic Look at Large Language Models
39mUnlocking reasoning in Large language models