Apache Spark on Azure - solution (...)

on YouTube ↗44m

" summary

Covers building a scalable streaming data pipeline on Microsoft Azure using Apache Spark, with Apache Kafka as the ingestion layer and Spark Streaming for processing. Explains how to process large, semi-structured or variable streams (IoT-like sources) at scale on Azure and discusses the solution's advantages and disadvantages.

AI-written summary · about

" chapters (11)

1. 0:00Introduction
2. 1:12Internet of Things concept
3. 5:14Stream vs batch processing
4. 8:29Apache Kafka key features
5. 10:58Energy meter system example
6. 16:00Why choose the cloud
7. 22:04Azure Databricks on-demand cluster
8. 25:43Demo: IoT data producer
9. 32:21Demo: Databricks data consumer
10. 37:10Summary
11. 42:33Q&A

" related talks

• 35mArchitectures and Trends for Event Stream Processing on Azure with Open Source SoftwareI. Ekpo · ApacheCon@Home 2021
• 48mStreaming Azure ServicesPedro Mendes, Miguel Jesus · Global Azure Portugal 2021
• 43mAnalyzing Big Data in Azure cloud using Spark and DatabricksMaria Zakourdaev · SQLBits 2023
• 20mGetting Started with Event Stream Processing via Apache Flink on Microsoft AzureIsrael Ekpo · ApacheCon@Home 2021
• 42mBuilding Robust Streaming Data Pipelines with Apache SparkZak Hassan · Open Source Summit North America 2017
• 52mSolving Analytical Problems using Apache SparkRohit Bhardwaj · Developer Summit 2020
• 26mBig Data on Azure with Structured, Unstructured and Streaming DataGustavo Ayub · Big Data Week São Paulo 2017
• 36mHow to Build a Solution Processing Terabytes of Data (...)?Kamil Dworak · 4Developers 2019