How to build your own Tiny Language Model from scratch
summary
Shows how to train tiny autoregressive models using the llm.c C/CUDA reference implementation and a fork with a runnable notebook that trains on CPU at ~10,000 tokens/sec. Claims reproduction of GPT‑2 for ~$20 cloud cost and notes Microsoft’s F4 (14B) matches GPT‑3.5 metrics. Summarizes datasets and findings: TinyStories yields coherent English with models as small as 30M parameters; curated FineWeb/edu data gives large performance jumps; mentions SmallLM family (135M) and practical tokenizer/training choices.
AI-written summary · about
chapters (5)
- 0:00Introduction
- 0:29Why Tiny Models Matter
- 17:35Building a Tiny Model
- 29:45Practical Implementation
- 35:46Q&A and Conclusion
related talks
28mFine-Tuning Small Language Models with Python
52mUnleash the Power of Large Language Models (LLMs)
19mSystem for Scaling Large Language Models
39mCollect Data to Fine-Tune Your Large Language Model
01h 04mA Very Small Language Model
18mHow to Make Small Language Models Work
43mBeyond the Hype: A Realistic Look at Large Language Models
01h 00mInside GPT – Large Language Models Demystified