Async Inference at Netflix: Batch and Near-Real-Time Serving
summary
Netflix runs a unified async inference platform that executes the same pipeline code in two modes: ephemeral Ray Data jobs for high-throughput batch and warm actor pools for near‑real‑time with scale‑to‑zero and per‑model queues. Orchestration and durability use Temporal, compute runs on Marco, and runtimes use vLLM/vLLM Omni with a custom Ray diffusion executor. The executor delivers 3.2x single‑GPU / 10.5x 8‑GPU speedups, and the stack Ray + vLLM + Marco + Temporal supports dynamic batching, priority preemption, and durable long‑running workflows.
AI-written summary · about
chapters (17)
- 0:00Introduction and overview
- 1:40Sync vs. async inference
- 4:05Async inference examples
- 5:21Architecture overview
- 7:58Platform stack components
- 9:26Shared pipeline substrate
- 12:35Compute and runtime
- 14:02Batch execution strategies
- 16:47Near-real-time inference
- 18:18NRT worker architecture
- 20:55Pipeline trace execution
- 22:22Inference runtime engine
- 23:34Multi-node inference
- 24:23Performance benchmarks
- 24:37Key takeaways
- 25:25Future roadmap
- 36:02Q&A session
related talks
38mNetflix's Ray Platform: From Deep Learning to GenAI
32mAnyscale's Ray Data: Revolutionizing Batch Inference
31mFaster Model Serving with Ray and Anyscale
22mJustin Basilico, Research/ Engineering Manager, Netflix
29mBuilding Netflix - from movie magic to ML reality
45mNetflix's success: Combining collaboration, hardware monitoring & AI
04mInterview
26mEfficiently serving LLMs at scale.