Improving open language models at a practical size
summary
Gemma 2 ships as 2B, 9B and 27B models and claims top-in-class size-normalized performance by training small models against teacher probability distributions instead of raw next-token labels, plus Transformer tweaks and careful data filtering. A 27B model trains up to ~13T tokens while 2B models hit capacity around ~8T, and the 2B runs on-device. The release includes teacher-guided pretraining, post-training alignment with teacher correction, safety filters and deployment tooling, and all models are open-sourced.
AI-written summary · about
chapters (14)
- 0:00Google invests in open models
- 1:05Designing for different platforms
- 2:30Model training phases
- 3:54Pre-training in practice
- 5:20Truth vs training answers
- 6:44Teacher models for training
- 7:46Token counts for model sizes
- 9:11Post-training with teachers
- 9:54Benchmark performance results
- 11:19Model safety in training
- 13:03Safety versus usefulness
- 14:06Safe model deployment tools
- 15:09Gemma impact and adoption
- 16:35Key use cases and availability
related talks
10mGemma 3 architecture and design
25mTraining Large Language Models in the Open
16mOpen models in the Gemini era
10mAI for everyone with Gemma
10mDemo: Using Gemma with the Hugging Face ecosystem
18mSmolLM: The Rise of Smol Models
08mLeveraging large-scale models for efficient learning
39mFlutter Project with Gemma: A State-of-the-Art Open Model