The mesh inside the mesh
summary
Migrates a monolithic ETL into a domain-oriented decentralised design that processes research data harvested from over 2,000 repositories, producing tens of millions of records in a semi-streaming pipeline. Builds a “mini mesh inside the mesh” by splitting one corpus into domain‑aligned and cross‑domain products, implementing three‑tier models, interdependent DAG flows, and reproducible records. Uses Delta Lake + Spark on an AWS-backed lakehouse (S3, EC2, managed services) with flow orchestration, and reports faster update cycles and lower infrastructure costs.
AI-written summary · about
chapters (11)
- 0:00Introduction and company background
- 6:50What is data mesh
- 11:07Transition to mini mesh
- 13:59Implementation approach and principles
- 17:29Lake house architecture zones
- 20:23Domain aligned data processing
- 23:15Cross domain reference data
- 25:46Visual clarity in architecture
- 26:08Guidelines from experience
- 29:20Technologies and tools used
- 31:11Closing and Q&A
related talks
39mData Mesh: Why, What and How
48mBuilding a data mesh
30mFrom Data Lakes to Data Mesh - The Evolution of Redmesh
48mData Mesh Paradigm Shift in Data Platform Architecture -
17mData Mesh: The Speedrun
57mOur DataMesh Journey: data architectures for microservices
20mEmbracing Data Mesh: A Modern Strategy for Scalable Data Management
47mModern Data Architecture: Breaking the Big Data Monolith