Uber's GenAI Leap: Batch Predictions Using Ray and vLLM
summary
Uber integrates Ray + vLLM on Michelangelo to run large GenAI batch prediction at scale, replacing Spark for GPU-heavy workloads. The system handles millions (up to ~50M) rows with 1k–5k input tokens and hundreds-to-1k output tokens, uses node-local single-copy model downloads and whole-node allocation to avoid GPU fragmentation, and distributes work across actors. Benchmarks show about 500 tokens/sec per GPU (batch 64) and they run typical daily jobs on ~80 H100s for 50M rows, finishing within a day.
AI-written summary · about
chapters (7)
- 0:00Introduction to GenAI batch predictions
- 3:59Batch prediction use cases at Uber
- 10:18Online-offline consistency challenges
- 13:51Leveraging Ray and vLLM for batching
- 19:35Benchmarking and performance results
- 22:05Key challenges and optimizations
- 23:32Summary and Q&A
related talks
33mScaling PyTorch Training with Ray Data on Uber's Michelangelo AI Platform
17mKeynote: Machine Learning & A.I. At Uber
19mMichelangelo ML Platform at Uber: Past, Present and Future
25mAccelerating ML at Uber with the Palette Feature Store
28mScaling Multimodal Data and Reinforcement Learning with Ray
26mUber’s Intelligent Insights Assistant
44mPanel: ML Platform on Ray
25mSponsored Session: Accelerating GenAI Inference: From AWS Deep Learning...