Keynote: Open-source Multimodal AI in the Wild
summary
Open-source multimodal stacks run end to end on Hugging Face with one-line pipelines and support for 4bit and 8bit quantization so models fit local hardware and preserve privacy. Examples: CLIP-style encoders plus LLM projectors (LAVa, Qwen-2-VL) enable zero-shot VQA and image search; OWL v2 and GroundingDINO deliver state‑of‑the‑art open‑vocabulary detection; combining OWL with Segment Anything produces OWL-SAM and SAM2 adds a memory/masket for tracking. About 593 Vision language models sit on the Hub ready for fine‑tuning with Transformers, LoRA and TRL/DPO.
AI-written summary · about
chapters (19)
- 0:00Multimodal AI introduction
- 4:00Open-source multimodal models
- 7:34CLIP and SigLIP for alignment
- 10:00OWL for zero-shot detection
- 11:24Segment Anything Model
- 12:55Vision language models
- 14:26How LLaVA revolutionized VLMs
- 16:13PJama’s open vocabulary segmentation
- 17:37Qwen-VL for multilingual OCR
- 18:40Document AI with retrieval VLMs
- 20:10Finding and evaluating open VLMs
- 22:44Deploying VLMs with TGI
- 23:49Fine-tuning VLMs with Transformers
- 26:20Quantizing models with bitsandbytes
- 27:50Parameter-efficient fine-tuning with LoRA
- 29:38RLHF for VLMs with TRL
- 31:27Quantization libraries for VLMs
- 31:48Small Vision for prototyping
- 32:11Q&A on multilingual and bias
related talks
33mKeynote: Fine-tuning our way towards openness in AI
14mKeynote: Building AI with Open Source & Hugging Face
16mKeynote: Open Source AI Everywhere- 14mKeynote: Building AI with Open Source & Hugging Face
27mThe Cutting Edge of AI: From LLMs to Multimodal Intelligence
21mMultimodal Open Source at Kyutai, From Online Demos to On-Device
35mOpen-source AI: why it matters and how to get started
13mKeynote: To Nemotron and Beyond: Building the Open Source AI Ecosystem