Understanding and Applying Self-Attention for NLP
summary
Claims that Transformer-style self-attention encoder replaces LSTMs for relation extraction and yields a new state of the art: experiments use Python with TensorFlow/PyTorch blocks, multi‑head attention, positional encodings and warm‑up learning rates, train on a 120,000‑example dataset with 42 relation classes and avg sentence length ~40, and report beating a previous 65.1 F-score baseline by about 0.5%. Key engineering details: relative positional embeddings, swapping layernorm for batchnorm, changing residuals and activation, convergence in ~60 epochs on a 6GB GPU.
AI-written summary · about
chapters (11)
- 0:00Self-attention in NLP
- 1:50Neural NLP model evolution
- 4:40Attention in sequence models
- 8:32Pure self-attention
- 12:05Positional encodings
- 14:57Multi-head attention
- 19:39Implementing self-attention
- 22:11Self-attention for relation extraction
- 26:53Modifying for relation extraction
- 30:25Q&A
- 35:40Future of self-attention
related talks
26mA Shallow Introduction to Self-Attention
01h 25mPaper Discussion: Attention is all you need
01h 10mIntroduction to Transformers
24mRevealing the Unseen: Visualizing Attention in Transformer Models
28mEverything you need to know about Transformer Models
23mUnveiling the Journey of NLP
01h 29mBuild Your Own Transformer
52mTowards fast and coherent long-form text generation