Uber's Journey in Building Multi-Modal Foundation Models
summary
Uber builds an in‑house encoder‑decoder multimodal foundation model trained on about 50 million documents across 120 document types to automate driver onboarding, receipts and name matching, using OCR plus image patches and a spatially aware attention block. Training runs on distributed Ray clusters with DeepSpeed and mixed precision, and a ~0.5–1B parameter sweet spot balances latency for online serving. Internal benchmarks show the model beats GPT and humans on extraction tasks and is deployed in production with caching and deterministic shuffling to scale.
AI-written summary · about
chapters (7)
- 0:00Introduction and agenda
- 0:33Project motivation
- 1:48Architecture and training
- 13:13Model evaluation
- 14:24Live demo and use cases
- 18:07Summary and conclusions
- 19:05Q&A session
related talks
17mKeynote: Machine Learning & A.I. At Uber
31mDeep learning for self-driving vehicle technology
18mAI Platform Q&A
13mPyTorch at Uber
26mUber’s Intelligent Insights Assistant
01h 23mMultimodal Deep Learning in the Real World
15mAdvancing Search and Observability at Uber with OpenSearch
18mAdvancing Search at Uber with OpenSearch