Accelerating Explorations in Vision and Multimodal AI Using Pytorch...
summary
TorchMultimodal adds native diffusion primitives so developers can build a Dolly 2 generator in five lines using an ADM U-Net plus stackable adapters like classifier-free guidance and inpainting, a 1000-step linear Gaussian schedule and a DDPM sampler; components run with FSDP at scale and torch.compile. Domain libraries ship BLIP-2 style alignment via a query transformer for two-stage multimodal LLMs, MAE Vit-large audio/image checkpoints and a TorchVision transform v2 40% faster pipeline that handles boxes, masks and videos via tensor subclasses.
AI-written summary · about
chapters (4)
- 0:00Diffusion models in PyTorch
- 8:03Multimodal LLM development
- 13:43Self-supervised learning
- 19:11TorchVision V2 transforms
related talks
12mIntroduction to TorchMultimodal
25mHow Modern PyTorch Supercharges Multimodal Training and Inference at Luma AI
28mLarge Multimodal Models: Understanding the State of the Art in Compute
40mKeynote: Open-source Multimodal AI in the Wild
40mThe Marvelous Magic of Multimodal AI
15mMulti-Modal AI Optimization: Combining Text, Vision, and Speech Efficiently
11mUnlock the Future of Generative AI: TorchTitan's Latest...
18mDISTMM: Accelerating Distributed Multimodal Model Training