Building data mesh on Databricks and Azure for a large manufacturing company
summary
A data mesh for a large European manufacturer runs on Azure ADLS + one Databricks workspace per product and a separate storage account per product, with Unity Catalog catalogs bound to specific workspaces for isolation. Bootstrapping uses a cookiecutter template (17 answers) and a hello‑world ETL deployed via Databricks Asset Bundles through dev/qa/prod targets and CI. Governance auto‑checks Terraform plan JSON with Open Policy Agent + Rego unit tests (TDD). One product is live, onboarding and manual Unity Catalog metastore steps remain due to Azure limits.
AI-written summary · about
chapters (6)
- 0:00Introduction to project
- 1:35Data mesh principles
- 4:05Technical stack and infrastructure
- 10:38Data product developer journey
- 20:39Automated computational governance
- 24:29Learnings and challenges
related talks
17mData Mesh: The Speedrun
44mData Mesh: Lessons from the trenches
48mBuilding a data mesh
19mBuilding out Tailwind Traders Data Mesh
23mApplying Data Mesh to Your Organization
31mData Mesh In Practice – How To Set Up A Data Driven Organization
52mElevate your data and AI governance with Databricks Data Intelligence Platform
30mData Mesh – Architecture And Insights From The Self-service Platform Boosting Financial Services