Building highly scalable IoT streaming solutions using Azure Event Hubs and MongoDB

on YouTube ↗47m

" summary

Shows a concrete pipeline: MongoDB’s Kafka connector reads change streams from Atlas and writes to Azure Event Hubs, then a sync task consumes the hub back into an Atlas collection. Demo uses a Python producer for fictitious stock ticks, Docker-hosted Kafka Connect (Azure preview), topic creation, publishFullDocument=true, and REST POST connector configs to achieve a round trip. Highlights operational fits: Kafka Connect preview, Atlas online archive with federated warm+cold queries, and real customers with 25 terabytes from 6,000 turbines.

AI-written summary · about

" chapters (15)

1. 0:00Intro
2. 3:09Azure lot Hub vs Azure Event Hubs
3. 4:46What is MongoDB?
4. 5:17Modelling a Basic "Thing"
5. 6:16Strict Schema vs. Flexible Data Model
6. 7:22Extensible at Runtime
7. 11:03MongoDB & MongoDB Atlas
8. 12:05Demo: Passing data through Azure Event Hubs
9. 18:52Getting data to MongoDB
10. 24:23What is Kafka Connect?
11. 25:30MongoDB Kafka Connector
12. 27:03Configuring Kafka Connect
13. 35:00MongoDB within Azure IoT Ref Architecture
14. 40:14MongoDB Atlas Online Archive
15. 44:56Summary

" related talks

• 26mConnecting distributed data with Apache KafkaJacob Bogie, TJ Laher · Azure Cosmos DB Conf 2022
• 14mSolving Data Engineering ChallengesSteve Lee, Joel Roland · DataEngBytes 2020
• 26mConnecting distributed data to Azure CosmosDB with Apache KafkaJacob Bogie, TJ Laher · Azure Cosmos DB Conf 2022
• 15mAzure IoT from the edge to ADXJohan Ludvig Brattås · SQLBits 2023
• 03h 00mStream Processing, Going Beyond Pub/Sub with Kafka Streams and...F. Ramière, D Petisme · Devoxx France 2021
• 34mBuilding and Scaling Robust Zero-code IoT Streaming Data PipelinesPaul Brebner · ApacheCon@Home 2021
• 53mApache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline!Viktor Gamov · Jfokus 2019
• 40mA Data Mesh Of StreamsBen Stopford · State of Data Mesh 2022