Scaling AV Simulation with Temporal
summary
NVIDIA runs millions of AV simulations per day on GPUs using a purpose-built TestStudio that uses Temporal for orchestration. The system scales from an initial 5,000-run limit to 4 million simulations by adding a lightweight batch manager, segregating work into network workers, light/heavy/build pools, and autoscaling workers. Large file transfers stream through dedicated workers, builds replace Jenkins with a single-concurrency Bazel worker and remote cache, results land in a data lakehouse, and task queue priority and fairness enforce SLA weighting.
AI-written summary · about
chapters (16)
- 0:00Introduction to AV simulation
- 1:53Demo of NVIDIA AV in action
- 3:42Challenges in AV verification
- 6:11AV simulation testing methods
- 9:01Scaling AV simulation demand
- 10:24Introducing Test Studio platform
- 11:47Temporal as workflow glue
- 13:11Temporal benefits overview
- 14:12Horizontal scaling with Temporal
- 15:18Scaling from 5K to 4M tests
- 17:25Admission control with signals
- 18:29Coding failures do not exist
- 19:55Worker resource optimization
- 23:01Task queue priority and fairness
- 24:27AI skills in workflows
- 25:32Q&A session begins
related talks
37mAsync Inference at Netflix: Batch and Near-Real-Time Serving
06mRTC @Scale, Video Session
14mFrom Reservation to 1,600 GPUs: A Petabyte-Scale Ray Data Pipeline
42mUsing Temporal Workflows for Seamless Dev + Research Collaboration & Continuous Algo D
21mHow Ray and Anyscale Make it Easy to do Massive-scale ML on Aerial Imagery
33mFrom Prototype to Production: Making Agentic AI Systems Reliable
14mBlitzScale: Fast and Live Large Model Autoscaling with O(1) Host Caching
45mNetflix's success: Combining collaboration, hardware monitoring & AI