Best practices for implementing a modern data architecture with Azure
summary
600 RSVPs and ~250 attendees; practical Azure stack guidance: use store all sources in Data Lake as the single hub, put streaming through Event Hubs + Stream Analytics and batch through Data Factory, run MPP and semantic queries in Synapse and ML/ETL in Databricks, secure secrets in Key Vault and monitor costs with Azure Monitor. Adopt a lambda hot and cold paths pattern, enforce zones (raw/stage/curated/sandbox), automated tests, and CI/CD with Azure DevOps for reproducible deployments.
AI-written summary · about
chapters (19)
- 0:00Introduction
- 2:35Presentation Start
- 5:45Modern Data Architecture
- 10:00Challenges
- 13:16Data Sources
- 18:29Lambda Architecture
- 24:15Orchestration DevOps
- 25:11Platform Practices
- 29:58Governance and Data Ops
- 33:03Data like best practices
- 35:38Data Ops best practices
- 38:32Data logic tests
- 40:52Branching merging strategy
- 42:14Automating delivery
- 43:30Modularization
- 44:08Data governance
- 45:08Data quality
- 46:10Security and access
- 47:06Conclusion
related talks
55mDesigning performant and scalable data lakes
19mHow to Architect your Data Platform for Success
56mDataDevOps for the Modern Data Warehouse on Microsoft Azure
55mBuilding Modern Data Warehouse with Azure Synapse Analytics!
40mData Lake House in the Cloud: The Azure Solutions
26mBig Data on Azure with Structured, Unstructured and Streaming Data
51mIntroduction to Azure Data Platform
43mAnalyzing Big Data in Azure cloud using Spark and Databricks