Implement a Serverless Data Platform with Microsoft Azure
summary
Shows an implemented Azure pipeline that actually moves events from Event Hubs into DataLake Storage, uses Stream Analytics to aggregate into partitioned Parquet files, and then queries and transforms them with Synapse serverless SQL pool and Spark pools. Describes tradeoffs: mixed batch+streaming for late arrivals, Event Hubs offsets and checkpointing, Service Bus/Event Grid differences, and practical choices like pipelines, data flows and creating gold tables (aggregated cost by country/account) that are recreated and validated.
AI-written summary · about
chapters (11)
- 0:00Serverless data platforms intro
- 3:17Data platform problem space
- 8:18Solution space & building blocks
- 12:14Streaming vs batching
- 19:53Relational vs data lake storage
- 25:00Demo: Ingest with Event Hubs
- 29:42Streaming with Stream Analytics
- 31:28Exploring with Synapse serverless
- 34:21Pipelines & Spark processing
- 36:32Demo: Transforming to gold dataset
- 41:57Key takeaways
related talks
52mImplement a Serverless Data Analytics strategy
47mImplement a Serverless Data Analytics strategy with Microsoft Azure
01h 01mServerless Azure Data handling
30mProcess large-scale data and computational function in serverless manner
47mBuilding Data Lake Platform Fully Serverless
54mUnderstanding the Serverless platform in Microsoft Azure
01h 01mServerless Architecture With Azure
42mBuild and automate a modern serverless data lake on