SmolLM: The Rise of Smol Models
summary
Claims that compact models match much larger ones by training choices and data curation: SmolLM family has 100M, 300M, 1.7B variants, pretraining mixes curated corpora (FineWebEdu, StackEdu, math) for 11 trillion tokens and runs 24 days on 256 H100s. Deployments show browser and on‑device use via Transformers.js, Llama‑cpp and MLC, fits on recent iPhones, supports instruction tuning, function calling and structured JSON outputs, and yields MMLU and extraction results close to much bigger models.
AI-written summary · about
chapters (7)
- 0:00SmolLM intro
- 2:45Why LLMs are big
- 6:14Scaling laws for small models
- 9:46SmolLM models release
- 12:07Small model use cases
- 14:18SmolLM training details
- 17:06Q&A
related talks
22mSpecializing Small Language Models with less data
16mExploring the world of Small Language Models
38mMeta's Roadmap for Full Stack AI
40mSurviving the Agentic Hype with Small Language Models
47mLaptop-sized ML for Text, with Open Source
30mLearning to wield large language models
18mImproving open language models at a practical size
58mFederated Open Language Models: Training LMs on Distributed Data