Transfer Learning - Entering a new era in NLP
summary
Transfer learning is now practical: BERT and Transformer variants move from papers to production, Google reports BERT serving ~10% of US search queries, and leaderboards jump for QA. Workflow is concrete — pretrain on ~30GB (Wikipedia + books), optionally domain adapt on ~5GB, then fine‑tune with start/end span head for extractive QA. Tooling is real: Hugging Face transformers and FARM run on GPUs, use smaller models/sequences for speed, and often get surprising gains with only hundreds to thousands of labels.
AI-written summary · about
chapters (19)
- 0:00Transfer learning in NLP
- 2:50Why transfer learning transforms NLP
- 5:40History of transfer learning
- 7:08Breakthrough language models 2018-2019
- 9:40Impact on ML engineering
- 11:24Three phases of transfer learning
- 12:30How masked language models work
- 16:28Language model variants
- 17:14Pre-training details and tips
- 20:49Domain adaptation for language
- 23:39Fine-tuning and applications
- 25:03Emerging model architectures
- 27:11Efficient smaller models
- 27:58Multilingual models beyond English
- 29:09Question answering applications
- 33:17How extractive QA models work
- 36:11Frameworks for transfer learning
- 38:22Demo: training QA model
- 39:09Summary and future impact
related talks
44mTransfer learning in NLP
23mUnveiling the Journey of NLP
58mAn Introduction to Transfer Learning and Hugging Face
32mAn Introduction to Transfer Learning in NLP and HuggingFace
40mPanel discussion "NLP"
43mThe Evolution of Domain Transfer in the Era of Pre-trained Language Models
41mBetter Machine Learning with Less Data
41mRepresentations from natural language data: successes and challenges