Connecting distributed data to Azure CosmosDB with Apache Kafka

on YouTube ↗26m

" summary

Demo shows Confluent Cloud using Apache Kafka and ksqlDB to join clickstream and user tables into curated topics and push results into Azure Cosmos DB Sync Connector with a fully managed setup. Schema Registry enforces Avro/JSON contracts and the connector uses Kafka metadata for unique IDs (topic+partition+offset). The team provisions the connector from the UI, it catches up to existing messages and writes documents into the Cosmos container, demonstrating end to end enrichment and successful sync.

AI-written summary · about

" chapters (10)

1. 0:00Introduction
2. 0:47Changing the world
3. 4:20Integrations with Azure
4. 11:48Demo Description
5. 15:03Topics
6. 15:48Schema Registry
7. 17:53Data Integration
8. 18:28Source Connectors
9. 19:38Connecting to CosmosDB
10. 25:18Wrap up

" related talks

• 14mPower real-time data streams from anywhere to Cosmos DB with ConfluentFlorence Sullivan · Microsoft Build 2023
• 41mApache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!Robin Moffatt · All Things Open 2020
• 49mLet’s Build a Streaming Data Pipeline!Robin Moffatt · DataXDay 2020
• 40mA Data Mesh Of StreamsBen Stopford · State of Data Mesh 2022
• 44mNo More Silos: Integrating Databases and Apache KafkaRobin Moffatt · code.talks 2019
• 47mBuilding highly scalable IoT streaming solutions using Azure Event Hubs and MongoDBRobert Walters · Global Summer 2020 Azure Data Fest 2020
• 29mApache Kafka and ksqlDB in Action : Let’s Build a Streaming Data PipelineRobin Moffatt · DevOps Pro Europe 2021
• 01h 03mApache Kafka Event streaming platform for .NET developersViktor Gamov · NDC Sydney 2019