Talk by Pranjal Awasthi (Google)
summary
Shows that adding an successor auxiliary task (multitask training with two heads, alternating gradient steps) makes decoder-only transformers trained on sequences up to length 20 generalize reliably to length ≈100 (about 5×) for sorting and improves robustness on addition, parity and repeats. Empirical probe of encoder/decoder embeddings finds an identity+successor primitive; auxiliary training strengthens that primitive. Proposes an automated random deletion D' construction for auxiliary distributions and gives a linear-scaling generalization theorem for marginled, depth‑1 transformers.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 3:12Length generalization in transformers
- 7:08Auxiliary tasks for length generalization
- 13:07Transformer embeddings for sorting
- 19:45Theoretical insights and automated tasks
- 29:55Limitations in length generalization
- 34:30Q&A
related talks
18mGuiding machine design using insights from simple sandboxes
01h 10mIntroduction to Transformers
56mTowards sequence-to-sequence models without activation functions
38mHow Transformers Learn Causal Structure with Gradient Descent
01h 16mHow Do Transformers Work?
46mLearning Theory of Transformers: Generalization and Optimization of In-Context Learning
36mNeural Networks
26mA Shallow Introduction to Self-Attention