Data Mesh: The Speedrun
summary
Cloud-native Data Mesh implementation uses a hybrid orchestration pattern with a single Airflow orchestrator and dataset-trigger choreography to keep cross-team dependencies visible and actionable. Heavy transformations run on Databricks + Unity Catalog for Spark familiarity, lineage and access controls, while niche serving uses Azure SQL or Elasticsearch to avoid cost and latency blowups. Data quality runs on Great Expectations with contract thresholds that can mark a product invalid. A shared data product blueprint and delayed data catalog choice are practical lessons that speed delivery but require upfront governance.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:15Organizational independence
- 2:36Cross-organizational data integration
- 3:47Airflow orchestration
- 4:49Unity Catalog governance
- 5:32Great Expectations validation
- 7:00Two-tier storage strategy
- 8:28Lessons from data focus
- 9:30Data catalog importance
- 10:36Flexibility vs standardization
- 12:20Final architecture blueprint
- 14:10Q&A
related talks
31mData Mesh In Practice – How To Set Up A Data Driven Organization
20mEmbracing Data Mesh: A Modern Strategy for Scalable Data Management
47mData Mesh - a hot air balloon?
39mData Mesh symphony: Orchestrating a synchronized data and business strategy ensemble
44mData Mesh: Lessons from the trenches
29mBuilding data mesh on Databricks and Azure for a large manufacturing company
48mData Mesh Paradigm Shift in Data Platform Architecture -
57mOur DataMesh Journey: data architectures for microservices