Transfer learning in NLP
summary
Transfer learning uses large pretraining plus fast fine‑tuning and delivers state‑of‑the‑art gains: masked language modeling and autoregressive LM objectives power models like BERT, RoBERTa, GPT‑2 and XLM, and scale up to 8.5 billion parameters (Megatron). Practical tools are the Transformers library, TensorFlow/PyTorch hubs and GitHub checkpoints. Distillation produces compact models — e.g. 66 million parameters distilled model retaining about 95% of BERT performance — making deployment and inference feasible.
AI-written summary · about
chapters (21)
- 0:00Intro
- 1:42Overview
- 2:51What is Transfer Learning?
- 4:05Why Transfer Learning in NLP? (intuitively)
- 5:37Why Transfer Learning in NLP? (empirically)
- 7:54Training: Sequential Transfer Learning
- 9:47History: The rise of language modeling
- 13:16A State-of-the-Art Transfer Learning Model
- 16:17Model: Using a typical Transfer Learning model
- 17:54Model: From shallow to deep
- 18:10Model: the example of BERT
- 20:07Model: Adapting for target task
- 21:41Training: Adaptation on a text classification task
- 24:30Open-sourcing: practical considerations
- 26:11TensorFlow-Hub
- 27:48Hugging Face library with Transformers
- 30:34Larger models on larger datasets
- 31:34A trend for smaller models
- 32:49Smaller models: Distillation from large models
- 34:08Shortcomings of pretrained language models
- 34:40Shortcoming of language modeling in general
related talks
45mTransfer Learning - Entering a new era in NLP
32mAn Introduction to Transfer Learning in NLP and HuggingFace
23mUnveiling the Journey of NLP
27mEmbeddings, Transfer Learning and More: LLM Tailored to You
01h 38mTraining Large Language Models: Practices and Research Questions
43mThe Evolution of Domain Transfer in the Era of Pre-trained Language Models
28mFine-Tuning Small Language Models with Python
41mBetter Machine Learning with Less Data