The Evolution of Domain Transfer in the Era of Pre-trained Language Models
summary
Pre-training changes what models find hard: short tests with distracting sentences expose sensitivity failures that are largely fixed by data augmentation of the training set, while a 2019 domain-generalization shared task shows swapping BERT for a stronger pretrained model (e.g., XLNet) gives the biggest gains. Log-likelihood likelihood splits reveal long-tail examples are much harder, and CodeQL generation fails because queries are rare at pretraining time. Analysis of addition shows pretrained token embeddings contain Fourier periodicities that fine-tuning leverages.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 1:27Why models break
- 4:15Distribution shift vs difficulty
- 9:59Harmless distribution shifts
- 16:35Pretraining distribution shift
- 23:28Pretraining alters difficulty
- 35:18Open questions
- 41:30Summary
related talks
44mTransfer learning in NLP
45mTransfer Learning - Entering a new era in NLP
01h 38mTraining Large Language Models: Practices and Research Questions
40mPanel discussion "NLP"
01h 05mControlling distribution shifts in language models: a data-centric approach.
32mAn Introduction to Transfer Learning in NLP and HuggingFace
01h 16mGeneralization in natural language processing
41mRepresentations from natural language data: successes and challenges