Melanie Warrick, Deep Learning Engineer, Skymind.io
summary
Explains how attention lets models focus on parts of input rather than process everything, contrasting soft attention/backprop which uses the full input and remains differentiable with standard gradient descent, versus hard attention/Monte Carlo policy gradient which samples glimpses and uses policy gradients (REINFORCE Williams 1992). Gives the concrete example of a deep recurrent attention model on SVHN that couples a CNN glimpse with two RNNs (location and classification), showing similar accuracy with ~80–90% fewer FLOPs and image-size–independent parameter count.
AI-written summary · about
chapters (8)
- 0:00Start
- 0:09Melanie Warrick
- 2:41Attention Model
- 8:10The Differences between Soft and Hard Models
- 12:11Hard Models
- 12:48Convolution Nets
- 14:10Recurrent Nets
- 15:06General Goals
related talks
42mAttention and Memory in Deep Learning Networks
01h 25mPaper Discussion: Attention is all you need
26mA Shallow Introduction to Self-Attention
34mTalk by Mahdi Soltanolkotabi (University of Southern California)
36mML Efficiency for Large Models: From Data Efficiency to Faster Transformers
01h 30mToward Understanding In-context Learning
51mDeep Learning - What is it and What It Can Do For You
27mUnderstanding Deep Neural Networks