Streaming: Real-Time Architecture for Connected Products in Automotive
summary
Built a real-time streaming service to meet the EU Data Act using Kafka compacted topics for configs, Kinesis for upstream ingestion, DynamoDB for lookups, a filter service that emits per-attribute messages, and Rabbit MQ with MQTT plugin as the public broker plus an activity tracker and config manager. Production reveals scaling surprises from ~14k to 14 million messages and forced running Rabbit MQ in a 3‑node setup, custom proxy and automation for costs and observability.
AI-written summary · about
chapters (13)
- 0:00EU Data Act introduction
- 6:28Project timeline
- 9:21Streaming solution key traits
- 10:23Architecture requirements
- 12:50Service flow details
- 17:44End-to-end streaming
- 19:09Live demo
- 20:34Implementation challenges
- 23:49RabbitMQ & IDP setup
- 25:39Proxy protocol hurdles
- 26:42Production scaling costs
- 28:52Tech stack & monitoring
- 29:14Project prep steps
related talks
43mEvent Streams & Data Rapids: Harnessing the Full Power of Messaging Platforms.
30mEvent Streaming and the Future of Applications
06mPREVIEW: Real-time Stream Processing at Scale
39m10 years of Kafka. Tips, tricks and lessons learned
42mIt's like 10,000 streams when what you need is a queue
47mBuilding High Performance Real-time Stream Processing Microservices
54mMicroservices with Kafka
31mEvent Streaming on Rails