Industry NLP: Multitask & Continual Learning
summary
Compares BERT‑based baselines (single‑task and shared encoder) with adapters and hypernetworks using three classification and four NER datasets on a single Tesla P4; adapters are ~0.5% of BERT and cut training time and memory to under <1% parameters while adapter fusion slows inference; hypernetworks generate task‑specific adapters with L2 regularization on generated weights; multitask performance roughly matches STL, NER drops ~0.5–2% but classification suffers a 20–35 F1 drop under continual updates.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 0:49Multitask and continual learning motivation
- 2:39Need for efficient NLP models
- 4:21Data and experimental setup
- 6:05Baselines and shared encoder
- 7:54Adapter-based multitask learning
- 11:05Hypernetworks for parameter sharing
- 14:15Results and performance
- 18:12Continual learning outcomes
- 21:20Key takeaways
- 23:29Q&A
related talks
40mPanel discussion "NLP"
22mEnterprise NLP with BERT | Optimized Text Analysis Framework
23mUnveiling the Journey of NLP
44mTransfer learning in NLP
21mEfficientBERT: Trading off Model Size and Performance
45mTransfer Learning - Entering a new era in NLP
28mBuilding one multitask model to rule them all
25mAccelerate NLP training with Amazon SageMaker