Unraveling Multimodality with Large Language Models
summary
Claims that open-source LLaMA 70B (moving to LLaMA 3) can power a nurse-facing healthcare assistant, chosen by evaluating speed, cost, precision; code uses PyTorch, Hugging Face, Accelerate, GPUs (G5.12xlarge) and LangChain to orchestrate model selection across LLaMA, Falcon, LLaVA/LAVA and mRAW. Demonstrations include OCR and chart reasoning on prescriptions and screenshots, a PDF report generator combining image/text, latent diffusion image edits via latent diffusion models/Stable Diffusion for UI mockups, and an agent that books an exam returning a protocol number (81536) to show end-to-end action.
AI-written summary · about
chapters (16)
- 0:00Introduction to multimodality in large language models
- 2:11Defining multimodality and its real-world applications
- 3:14Foundation models for multimodality
- 6:04Healthcare model selection
- 8:54Comparing Lama 2 and Lama 3 in practice
- 10:38Beyond text with multimodality
- 14:12LLaVA and visual tuning
- 17:05Reading prescriptions
- 19:35Charts beyond OCR
- 21:40Stress analysis with multimodality
- 23:06Virtual try-on imaging
- 24:32Diffusion models explained
- 27:46Diffusion in healthcare
- 30:17Multimodel systems with LangChain
- 33:15Automating workflows
- 37:08Future of multimodality
related talks
33mLessons Learned Applying Large Language Models in Healthcare
43mLarge Language Models for Devs: from zero to your first LLM application
30mLearning to wield large language models
47mLaptop-sized ML for Text, with Open Source
36mProduction-Ready AI Agents: From LLMs to Small Language Models
31mDeploying Large Language Models with Ease
27mThe Cutting Edge of AI: From LLMs to Multimodal Intelligence
33mOpening Keynote: LLMs, between fantasy and reality - the poin...