Large Multimodal Models: Understanding the State of the Art in Compute
summary
Tracks the concrete shifts from CNNs to transformers and large multimodal stacks: ViT 16×16 patches replaces local convolutions, CLIP (400M image–text pairs) enables zero‑shot image–text retrieval, and segmentation now includes SAM v3. Architectures use a vision encoder plus a multimodal connector / Q‑former to map to LLM token space. Multimodal models power VQA, document understanding, visual search and UI navigation but are heavy, inconsistent on precise coordinates, and overkill for real‑time edge tasks where CNNs still win.
AI-written summary · about
chapters (10)
- 0:00🚀 Build Stuff Conference – Get Your Tickets!
- 0:30Introduction: Why Talk About Vision in the Age of LLMs?
- 3:00How Machines “See” Images
- 6:10CNNs: The Traditional Foundation of Computer Vision
- 10:00Common Vision Tasks: Classification, Detection, Segmentation
- 13:40Vision Transformers: Bringing Attention to Images
- 17:10CLIP and Contrastive Learning
- 20:30Large Multimodal Models: Architecture & Reasoning
- 24:30Practical Use Cases: VQA, Search, Documents, and Agents
- 28:10Limitations of Multimodal Models for Classic Vision Tasks
related talks
13mInside Multimodal Models
01h 23mMultimodal Deep Learning in the Real World
01h 06mMixed-modal Language Modeling: Chameleon, Transfusion, and Mixture of Transformers
27mThe Cutting Edge of AI: From LLMs to Multimodal Intelligence
26mMaximizing Multimodal: Exploring the search frontier of text-to-image models
40mKeynote: Open-source Multimodal AI in the Wild
15mMulti-Modal AI Optimization: Combining Text, Vision, and Speech Efficiently
12mEmergent Trends in Large Multimodal Models