From Data Lake Entanglement to Data Mesh Decoupling: Scaling a Self-Service Data Platform
summary
Claims conversion from a monolithic EDW to a domain-oriented data mesh using PySpark Delta Lake DQX and Unity Catalog rests on measurable changes: about hundreds of data products across 10 domains, an 80-person data org, templated CI/CD, a shared library, and perview catalog ingestion for lineage. Implementation details include per-product jobs and clusters, schema evolution policies, view-based projections to absorb changes, and table-triggered refreshes — concrete patterns that either scale or reintroduce coupling.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 1:17Fondolanda data platform
- 4:28EDW to data mesh journey
- 7:20Data mesh adoption reasons
- 8:43Early data product challenges
- 10:50Unity Catalog switch
- 11:52Change and scalability lessons
- 12:55Team and governance decoupling
- 15:43Shared libraries and standards
- 17:52Tagging policies for control
- 18:56DQX data quality checks
- 19:38Data contracts and schema evolution
- 23:09Runtime decoupling with jobs
- 24:59Lineage tracking and impact
- 26:47Time coupling challenges
- 28:33Summary
related talks
54mData Mesh in Practice
51mWhat is a Data mesh?
48mAn Evolution of Data Architectures - Lambda, Kappa, Delta, Mesh & Fabric
36mData Mesh: Kicking Off a Mentality Change in an Organisation
42mAn Evolution of Data Architectures - Lambda, Kappa, Delta, Mesh & Fabric
11mBreaking Data Silos: Enterprise Data Management
31mDelta Lake: Open Source Reliability and Quality for Data Lakes
20m3-phased approach to delivering a lakehouse with data mesh