Everything you need to know about Transformer Models
summary
Explains the actual mechanism: numpy implementation of scaled dot product attention and how queries, keys, and values are learned and computed; shows that attention is a weighted average with concrete weights and visualized attention weights for examples like “she took the train to Berlin”; notes BERT’s stack of 12 identical encoder layers and the practical benefit that attention enables parallel processing of sequence elements versus RNNs; demos using Hugging Face models with PyTorch or TensorFlow and what each example outputs.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:30Google Translate to Transformers
- 3:21Transformer architecture basics
- 6:14Word embeddings
- 8:02Attention mechanism
- 13:03Self-attention queries and keys
- 18:23BERT attention scores
- 19:49Parallelization in Transformers
- 22:25Q&A on attention
- 26:26Q&A and resources
related talks
01h 29mBuild Your Own Transformer
36mNeural Networks
26mA Shallow Introduction to Self-Attention
26mRethinking the Transformer: Toward Native Multimodal Architectures
01h 25mPaper Discussion: Attention is all you need
38mUnderstanding and Applying Self-Attention for NLP
43mVisual Attention is All You Need: Introduction to Visual Transformers
01h 26mFrom RNNs to Transformers: A Journey through Language Modeling