Everything you wanted to know about VLMs but were afraid to ask
summary
Overview of concrete capabilities: contrastive CLIP similarity embeddings power zero-shot queries; combine with Segment Anything to turn few box detections into pixel masks and run video tracking. Demonstrations use Florence 2 (small, 0.2B params) for captioning, region OCR and 90%+ accuracy on messy handwriting, and a 3B model fine-tuned to extract LaTeX from papers. Deployment uses Hugging Face weights in Google Colab with Transformers; single-frame latency drops from ~5s to ~400ms, approaching ~30 FPS for segmented tracking.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 1:46History
- 2:44Clip
- 6:34New VLMs
- 9:20Example
- 13:54Vision Language Model
- 14:33TaskSpecific Models
- 17:18Florence II
- 24:44Questions
related talks
32mTeaching Machines to See:Vision-Language Models Made Efficient with SmolVLM
23mVision-Language Models for Fraud Detection
46mML Use Cases at Think Big Analytics
26mMaximizing Multimodal: Exploring the search frontier of text-to-image models
49mHow grounded are ungrounded LLM models
25mLights, Camera, Inference! Video Generation as a Service With VLLM-O...
40mKeynote: Open-source Multimodal AI in the Wild
35mLarge-Scale Video Intelligence