Streaming ETL on the Shoulders of Giants
summary
Concrete recipe for moving events into a live operational store using configuration not code, connecting Apache Kafka to MongoDB via Kafka Connect and built-in SMTs to do filtering, keying and pre-aggregations. Demo runs Kafka plus a MongoDB replica set, posts a JSON connector config, writes hourly/daily rollups and a device status collection, then a deactivation toggles a flag that stops downstream generation — the pipeline works end to end. Connector is Apache v2 open source, now available in MongoDB beta and Confluent-verified.
AI-written summary · about
chapters (10)
- 0:00Start
- 11:07Persistence Layer
- 16:00Streaming Platform Kafka
- 25:50Change Data Capture
- 30:45Exemplary Use Cases
- 32:54Synchronizing Data across Services
- 38:07Source and Sink Connectors
- 40:11Sink Connectors
- 41:28Data Integration
- 44:31Live Dashboard
related talks
30mUsing MongoDB with Kafka - Use Cases and Best Practices
41mApache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
14mChange streams act on events in realtime
49mLet’s Build a Streaming Data Pipeline!
29mApache Kafka and ksqlDB in Action : Let’s Build a Streaming Data Pipeline
26mConnecting distributed data with Apache Kafka
32mDiscover the Modern Way to Build Real-Time Analytics Solutions at Any Scale
40mThe Changing Face Of ETL: Event-driven Architectures for Data Engineers