Mixed-modal Language Modeling: Chameleon, Transfusion, and Mixture of Transformers
summary
A unified early-fusion effort tests three concrete systems and training facts: Chameleon (tokenized images) converts images to ~1,000 discrete tokens and hits irreducible quantization loss and severe training instability at scale, Transfusion (diffusion-integrated) keeps images as ~64 continuous patches, trains stably (7B params, ~1T tokens, ~3.5B images), and scales far better for image quality, and mixture of transformers routes modality-specific parameters (MoT) to cut modality competition and improve image generation.
AI-written summary · about
chapters (10)
- 0:00Introduction and background
- 4:56Image generation paradigms
- 7:37Chameleon architecture
- 14:38Scaling and multitask tuning
- 23:17Scaling challenges and stability
- 33:14Limitations of discrete models
- 35:04Transfusion architecture
- 47:14Performance of Transfusion
- 52:49Mixture of Transformers
- 56:30Heterogeneous scaling and conclusion
related talks
13mInside Multimodal Models
28mLarge Multimodal Models: Understanding the State of the Art in Compute
26mRethinking the Transformer: Toward Native Multimodal Architectures
15mMulti-Modal AI Optimization: Combining Text, Vision, and Speech Efficiently
12mIntroduction to TorchMultimodal
49mBen Hall & Barbara Fusinska
43mEncounters of the Third Kind With the Multimodal Generative AI Revolution
51mLarge Language Models for Real-World Applications - A Gentle Intro