Compress language models to effective & resource-saving models with knowledge distillation
summary
Distills a Flair pipeline based on XLM-R teacher into an RNN student with GloVe for German NER, training the student on teacher-labeled data plus originals; student model runs on CPU/edge, trains in about five hours on a company GPU, yields roughly a 3% average performance drop while running ~40× faster and substantially smaller than the teacher, and is production-used in textwerk and several domain projects (environmental and legal) with manual eval and architecture tuning ongoing.
AI-written summary · about
chapters (12)
- 0:00Introduction and motivation
- 2:48AI model growth challenges
- 6:00Model compression methods overview
- 9:55Knowledge distillation explained
- 13:10Types of knowledge in distillation
- 15:42Training schemes in distillation
- 17:07Evaluating distillation results
- 19:38Distilling German language models
- 23:32Model selection pitfalls
- 24:55Future research directions
- 25:58Practical applications today
- 27:01Conclusion and Q&A
related talks
22mSpecializing Small Language Models with less data
08mLeveraging large-scale models for efficient learning
29mOptimizing Knowledge Distillation Training With Volcano
44mTransfer learning in NLP
18mHow to Make Small Language Models Work
43mDemocratizing NLP content modeling with transfer learning using GPUs
31mBuilding Large-scale Localized Language Models
42mPost-Training Small Language Models: the adventures of a practitioner