When Foundation Models Go Open: Unlocking the Promise From Infrastructure To Impact in Health
Talkat Open Source Summit Europe 2025 · on Wednesday, 2025-08-27· at 11:55· Elicium 1 (Elicium Level 1)·
summary
Claim: curated, high quality medical image text pairs beat larger noisy web scraped sets. Built OpenPMC 2.2M → 18M image text pairs with subfigure decomposition and in‑text summary augmentation, trained CLIP‑style encoders on 800 GPUs, and open sourced the subfigure extractor. Results show significantly different embeddings and superior performance on zero-shot retrieval and classification versus 15M+ datasets, with ablation proving subfigure extraction and in‑text summaries drive the gains.
AI-written summary · about
chapters (12)
- 0:00Open foundation models in healthcare
- 1:07Multimodal understanding vs models
- 6:59Vision language models data
- 8:02CLIP and contrastive learning
- 11:53Scaling medical data challenges
- 15:55Medical image-text pairs creation
- 21:12Open PMC dataset benefits
- 23:22Embedding representation differences
- 24:51Scaling to 18M curated pairs
- 26:37Why open medical AI matters
- 33:00Open infrastructure impact
- 34:05Question and answer session
related talks
26mMaximizing Multimodal: Exploring the search frontier of text-to-image models
25mGain full control of your apps with Open-Source AI
01h 08mGoing multi-modal
33mUnlocking Insights From Multimodal PDFs Using OpenSearch and Vision-Language Models
33mLessons Learned Applying Large Language Models in Healthcare
13mInside Multimodal Models
54mThe Future of AI: Building the Most Impactful Technology Together
31mText and Code Embeddings