Discovering the BERT family of models
summary
An overview of the BERT family, explaining that BERT uses Transformer encoders to create contextualized embeddings and is trained with Masked Language Modeling and Next Sentence Prediction on BookCorpus and English Wikipedia. It compares base (110M) and large (340M) model sizes and notes multi-day TPU training, then gives practical demos using TensorFlow/Keras Hub and Hugging Face: a DistilBERT classifier achieving ~84% accuracy on Disaster Tweets (160-token inputs) after two epochs, and a multilingual BERT for sentence-pair classification (entailment/contradiction/neutral) with runtimes of about 10 minutes on TPU versus ~1 hour on GPU.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 1:50BERT in Google tools
- 2:54Goals and demos
- 3:37Dense and RNN text models
- 6:25Text embeddings and semantics
- 9:32Transformers and attention
- 14:30BERT evolution and variants
- 16:59BERT architecture
- 19:46Training BERT and variants
- 22:59BERT MLM and NSP
- 24:42WordPiece tokenization
- 27:52Fine-tuning BERT
- 30:00Disaster Tweets dataset
- 31:46Tweet classification demo
- 36:28Sentence pair classification demo
- 41:52Summary and BERT future
related talks
41mRepresentations from natural language data: successes and challenges
57mNatural Language Processing 2020 Discussion
02h 50mUnderstanding and Using AI Language Models
32mAn Introduction to Transfer Learning in NLP and HuggingFace
02h 10mIntro to Hugging Face: Fine-tuning BERT for NLP tasks
01h 00mInside GPT – Large Language Models Demystified
47mInside GPT – Large Language Models Demystified
28m"Embeddings: How Computers Learned to Read"