Roblox's Journey to Supporting Multimodality on vLLM
summary
Roblox integrates multimodal inputs (image, audio, video) into vLLM and ships an end-to-end pipeline that is OpenAI Vision API compatible. Engineering delivers a registry-based input mapper to handle dynamic embedding lengths (examples range from 576 up to ~15,000 tokens) and an input processor for Hugging Face–style prompts. They enable precomputed embeddings, apply tensor-parallel encoders to avoid wasting >1 GB per GPU when encoders are replicated, and already supports over 10 multimodal models with Day‑One releases for Pixel and Llama 3.2. Future work targets image hashing, chunk prefill, and CPU preprocessing optimizations.
AI-written summary · about
chapters (12)
- 0:00Introduction to multimodality
- 0:50Speaker introduction and team overview
- 2:34Why support multimodal models on vLLM
- 3:38Brief history of multimodality support
- 5:24First PR and initial challenges
- 7:50Dynamic input shape challenges
- 10:59Designing the input processing framework
- 13:47OpenAI Vision API compatibility
- 15:52Performance optimizations for multimodal models
- 19:48Multimodal Llama 3.2 specifics
- 21:36Future roadmap and open questions
- 24:26Q&A session
related talks
27mThe Cutting Edge of AI: From LLMs to Multimodal Intelligence
40mKeynote: Open-source Multimodal AI in the Wild
43mWhat is multimodal RAG, and can we build a village with it?
13mTraining Embedding Model Resiliently for Multimodal M...
28mLarge Multimodal Models: Understanding the State of the Art in Compute
40mThe Marvelous Magic of Multimodal AI
25mLights, Camera, Inference! Video Generation as a Service With VLLM-O...
28mIndirect Prompt Injection Into LLMs Using Images and Sounds