Representations from natural language data: successes and challenges
summary
Overview of BERT as a bidirectional transformer pre-trained with masked tokens and a next-sentence objective, yielding state-of-the-art on SQuAD and many tasks; larger 24-layer models consistently beat 12-layer ones and adapter modules let production services reuse weights. Two case studies: a feed-forward language-ID plus a constrained decoder gives an 800x speed-up and a 19.5% absolute gain on code-mixed sets, and a released challenge set of 30,000 examples for noun–verb ambiguity plus contextual embeddings yields a 52% relative improvement and a 28% downstream error reduction.
AI-written summary · about
chapters (23)
- 0:00Intro
- 1:15Single model
- 2:06Bert
- 3:05Attention
- 4:58Attention in the transformer
- 8:18Next word prediction
- 10:49Non classification problems
- 12:11Pretraining approaches
- 13:31Pretraining tasks
- 15:53Pretraining recipe
- 17:56Results
- 19:31Why is this working
- 22:22Do I need full burped models
- 23:49Pretrained models
- 24:39Challenges
- 25:07Multiple languages
- 28:35Script correction
- 29:33Non verba ambiguity
- 30:46Data sets
- 31:57Accuracy
- 33:06Pronunciation
- 33:44Challenge sets
- 36:13Conclusion
related talks
43mDiscovering the BERT family of models
57mNatural Language Processing 2020 Discussion
50mKnowledge is embedded in language neural networks but can they reason?
45mTransfer Learning - Entering a new era in NLP
43mThe Evolution of Domain Transfer in the Era of Pre-trained Language Models
01h 00mInside GPT – Large Language Models Demystified
01h 26mFrom RNNs to Transformers: A Journey through Language Modeling
02h 50mUnderstanding and Using AI Language Models