Attention to Detail: Fine-Grained Vision-Language Alignment
summary
Presents concrete methods and results: adapts CLIP-style models into phrase-object alignment using Visual Genome (≈1,000 categories, millions of boxes) plus smaller phrase-grounding sets like Flickr30k (30k images), creates ~10,000 generated descriptive phrases and several million pseudo-labeled pairs, and shows a zero-shot model matches ≈10 shots of detector training. Introduces description fine-tuning that yields +10% AP on fine-grained retrieval and +4–5% on VQA benchmarks, builds a Hugging Face demo and an OpenVLThinker pipeline that, after alternating RL and behavior cloning, exceeds GPT-4V on select reasoning tasks.
AI-written summary · about
chapters (11)
- 0:00Vision-language alignment intro
- 1:25Fine-grained visual recognition
- 3:54Early visual-language methods
- 5:45CLIP limitations
- 9:20Phrase grounding for detection
- 15:10Descriptions for inference challenges
- 21:00Context-sensitive queries
- 27:46Visual reasoning failures
- 32:29Refining LLMs with details
- 37:09Reasoning in vision-language models
- 41:46Summary and future
related talks
32mEverything you wanted to know about VLMs but were afraid to ask
20mGenAI synthetic data used to fine tune a domain specific model
40mKeynote: Open-source Multimodal AI in the Wild
24mMulti-Modal AI Agents Explained | Open-Vocabulary Alignment
13mInside Multimodal Models
01h 04mUnderstanding and Steering Generative AI Systems
27mThe Cutting Edge of AI: From LLMs to Multimodal Intelligence
51mLarge Language Models for Real-World Applications - A Gentle Intro