Rethinking the Transformer: Toward Native Multimodal Architectures
summary
Argues for training on raw information rather than language proxies, proposing tokenizer-free transformer designs and hierarchical attention to handle wildly different granularities. Reports a proof‑of‑concept: a 600M model trained on raw BMP bytes (about 300 GB of data), 32k context, using a new lighthouse attention variant compatible with FlashAttention that yields ~4× training speedup; training runs one day on 64 B200s and unconditional byte generation produces valid BMP headers and low‑res images, with KV‑cache inference taking seconds. Challenges: byte‑level autoregression is slow, cross‑modality granularity and generation order.
AI-written summary · about
chapters (14)
- 0:00Rethinking transformers
- 1:03Scaling limits of transformers
- 2:27Multimodal model limits
- 4:55Information modeling
- 6:00Large information models
- 8:47Information modeling bottlenecks
- 11:11Raw data transformer
- 13:59Efficient attention for multimodal
- 14:41Slow inference and generation
- 16:06Raw image experiment
- 18:14Unconditional generation results
- 19:39Conditional generation
- 20:22Model applications
- 21:07Q&A
related talks
01h 06mMixed-modal Language Modeling: Chameleon, Transfusion, and Mixture of Transformers
41mBridging Pixels and Words: The Transformer Vision Revolution
15mBreaking the Layer Barrier: Remodeling Private Transformer Inference with...
38mFrom Von Neumann to Transformers: An Infra Engineer's Guide to What's Actually Inside an LLM.
34mBeyond Transformers - Intro to RWKV Architecture & The World To...
28mLarge Multimodal Models: Understanding the State of the Art in Compute
01h 26mFrom RNNs to Transformers: A Journey through Language Modeling
01h 16mHow Do Transformers Work?