Multimodal Deep Learning in the Real World
summary
Claims that multi‑modal models are ready for production meet concrete caveats: OCR + Gemini notebook works for simple scanned pages and returns usable summaries, but larger models like ID‑Effects 8B need ≥32 GB GPU (A100) and still hallucinate on complex extraction. Time‑series + imagery is promising—CrossViVit / EarthFormer demonstrate real improvements—yet dataset alignment often fails (archival Hebrew images vs text), fine‑tuning is costly, and production requires GPU clusters, Kubernetes, Cloud Functions/PubSub and careful retrieval/embedding checks.
AI-written summary · about
chapters (16)
- 0:00Multimodal deep learning intro
- 3:16Value of multimodal systems
- 7:51Building multimodal challenges
- 11:27Key implementation questions
- 13:33Recent multimodal research
- 17:06CLIP and embedding alignment
- 20:18Earthformer for satellite forecasting
- 23:13CrossVit for time series fusion
- 25:38Realm for healthcare records
- 28:07Historical document system
- 31:16OCR plus LLM processing
- 42:12IDEFICS for document analysis
- 49:46Multimodal embeddings with search
- 1:01:52Production system architecture
- 1:05:11Earthformer demo and challenges
- 1:10:52Q&A
related talks
28mLarge Multimodal Models: Understanding the State of the Art in Compute
27mThe Cutting Edge of AI: From LLMs to Multimodal Intelligence
01h 08mGoing multi-modal
40mKeynote: Open-source Multimodal AI in the Wild
37mA Practical Guide to Robust Multi modal Machine Learning in Education
12mEmergent Trends in Large Multimodal Models
15mMulti-Modal AI Optimization: Combining Text, Vision, and Speech Efficiently
21mHow Google DeepMind is researching the next Frontier of AI for Gemini