Data Platform Infrastructure at Rivian: A Deep Dive
summary
Rivian describes a production data platform that ingests vehicle telemetry via Kafka, processes streams with Flink, and serves time-series queries from Druid on Kubernetes (EKS) using MSK, S3 deep storage and operators. Concrete wins: switching CPUs to Graviton yields 20% processing cost cut, a hot service rewritten from python→rust cuts running pods to 10%, and aggressive bin‑packing with Carpenter for node autoscaling. Older Kafka data moves to S3, Druid uses tiered nodes and queries from deep storage, cross‑AZ traffic is minimized, and environment provisioning is automated with Crossplane + ArgoCD.
AI-written summary · about
chapters (15)
- 0:00Data platform intro
- 1:07Data flow architecture
- 3:37Real-time processing components
- 6:10Kubernetes and AWS scaling
- 7:37Apache Druid deep dive
- 10:03Cost optimization strategies
- 13:14Storage and networking costs
- 16:33Future scalability planning
- 19:02Infrastructure automation
- 21:30Q&A: Python to Rust
- 23:17Q&A: Self vs managed services
- 25:47Q&A: Kafka clusters
- 27:10Q&A: Crossplane vs Terraform
- 29:17Q&A: Flink vs Spark
- 29:38Closing remarks
related talks
24mCreating a Data Platform for Software-Defined Electric Vehicles
35mReal-time analytics at IoT scale
10mData Rivers — The New Analytics Architecture
34mTransforming Data Processing with Kubernetes: Journey Towards a Self-Serve Data Mesh
49mBuilding a Data-Driven Bank
45mAccelerated analytics: Building the next-gen data platform for Hertz
26mRe-architecting the Data Platform with Spark on Kubernetes
31mThe Future of Data on Kubernetes: From Database Management To AI Foundation