Towards sequence-to-sequence models without activation functions
summary
Argues that sequence-to-sequence models can drop element-wise activations and get expressivity from explicit polynomial expansions implemented with Hadamard product blocks and CP-style tensor factorizations. Shows image and generator models reach ImageNet-level accuracy and parity with transformers using no ReLU or GELU, often with half the parameters, and boosts degree exponentially via 2^m degree growth by composing second-degree blocks. Reports better extrapolation on digit-addition after a space-augmentation trick, but notes sample inefficiency and failure to learn a true algorithm for very long sequences.
AI-written summary · about
chapters (12)
- 0:00Activation-free sequence models
- 3:24Polynomial expansions replace activations
- 10:05High-degree polynomial networks
- 17:45Early image experiments and limits
- 21:44Activation-free model matches transformers
- 24:16Polynomial networks for sequences
- 29:29Sequence-to-sequence without activations
- 36:41Causal structure in polynomial models
- 40:20Training on three-digit addition
- 45:07Extrapolation and length tests
- 51:20Future directions
- 53:53Conclusion
related talks
40mAn Occam's Razor Principle for Transformers?
38mTalk by Pranjal Awasthi (Google)
34mThe physics of deep learning using tensor networks
01h 25mPaper Discussion: Attention is all you need
43mBeyond the Hype: A Realistic Look at Large Language Models
26mA Shallow Introduction to Self-Attention
52mGeneralization in the representations and computations of frontier language models.
25mEmbeddings, Transformers, RLHF: Three key ideas to understand ChatGPT