Architectures and Trends for Event Stream Processing on Azure with Open Source Software

on YouTube ↗35m

" summary

Survey of deploying Apache stream projects on Azure with concrete choices and gaps: offers Confluent Cloud or Azure Event Hubs (Kafka protocol) as fully managed Kafka, HDInsight as semi‑managed, and AKS/VMs with the Confluent Operator for self‑managed control; Spark runs on Azure Databricks or HDInsight; Flink, Beam, Storm and Pulsar run on AKS/VM scale sets. Connectors exist (Kafka Connect with Cosmos DB, Debezium, Event Hubs, Blob) and AAD/VNet secure traffic; work is active to add Flink blob/data‑lake support (streaming file sink in v1.14) and more Beam/Cosmos/Event Hubs integrations.

AI-written summary · about

" chapters (16)

1. 0:00Start
2. 4:31What is Event Stream Processing?
3. 6:51WHY OPEN SOURCE?
4. 7:50Open-Source Toolkits for ESP on Azure
5. 8:50Core Components of ESP
6. 15:03Apache Kafka
7. 18:28Apache Spark
8. 21:31Integration Options
9. 23:05Kafka Connect Data Sources
10. 24:13Kafka Connect Data Sinks
11. 25:00Spark Connectors
12. 25:28Integration with Apache Storm
13. 25:39Integration with Apache Flink
14. 26:24Integration with Apache Beam
15. 26:47Integrations with Apache Pulsar
16. 30:12Open-Source Toolkits on Microsoft Azure

" related talks

• 51mImplementing an Event Sourcing strategy on AzureOlena Borzenko, Eldert Grootenboer · Build Stuff 2021
• 44mApache Spark on Azure - solution (...)Kamil Dworak · 4Developers Katowice 2019
• 48mStreaming Azure ServicesPedro Mendes, Miguel Jesus · Global Azure Portugal 2021
• 43mAzure Messaging – Streams, Queues, and Pub SubClemens Vasters · bit summit 2025
• 51mLightup your Mobile Apps with AzurePraveen Srivatsa · Great Indian Developer Summit 2015
• 30mWhat’s new with Azure MessagingKevin Lam · Microsoft Build 2023
• 31mEvent-Driven Architecture - A successful strategy for a journey to the cloudMirna Alaisami · ContainerDays 2022
• 48mCloud Native Event Streaming with Kafka & Open StandardsKasun Indrasiri · GOTO Chicago 2023