Rivian: Scaling Data & ML Platforms
summary
Rivian describes building Polaris data platform to manage ADAS data that grows from 6B to 2T records/day and from 14 TB to 1.6 PB in six months, achieving reduced SLA from 150 to 4 hours. The stack uses S3, AWS Batch, EKS with EKS + KubeRay + Kueue for Ray-based training, MLflow for experiments, Grafana/Polaris Cosmo for observability, and RX CLI for job submission. Operational fixes include bundling jobs to beat AWS TPS limits, slimming Docker images, and fair-share queuing to avoid large-simulation hogging.
AI-written summary · about
chapters (9)
- 0:00Intro
- 2:55Data Volume
- 5:44Polaris Platform
- 12:51ML Infrastructure
- 15:07Engineers Interaction
- 17:43Grafana
- 18:29Cosmo
- 19:21Conclusion
- 20:51Questions
related talks
24mCreating a Data Platform for Software-Defined Electric Vehicles
35mReal-time analytics at IoT scale
45mAccelerated analytics: Building the next-gen data platform for Hertz
21mML Infrastructure for Autonomous Vehicles
17mR & Python playing nice, in Production
20mAccelerating AI pipelines w/ NVIDIA on AWS: An autonomous driving case
29mScaling ML Pipelines for Autonomous Driving with Ray
29mGet rolling with ML on AWS DeepRacer