The journey of Data Mesh and Compliance in a Multi-Regional Data Lake
summary
Built a customer-facing Atlassian Data Lake that actually serves product analytics, data exports and in-product dashboards, using logical CDC with Protobuf (outbox-style) to avoid DB-specific binlog coupling, processed via Flink + Delta + DBT where transformations are consumer-owned, and resolved with a last-write-wins versioning scheme. Deploys per-AWS-region and supports multi-region per-customer migrations by replaying raw Protobuf audit logs; Delta Sharing exposes exports. System is live and powering Atlassian Analytics and Delta-sharing exports.
AI-written summary · about
chapters (11)
- 0:00Data Mesh and compliance journey
- 1:05Customer-facing data lake motivations
- 3:32Logical vs physical replication
- 8:51Logical replication with Data Mesh
- 11:00Schema management and Proto Buffs
- 13:27Streaming architecture and real-time analytics
- 17:42Last-write-win strategy at scale
- 19:05Multi-region compliance necessity
- 21:12Multi-region data deployment architecture
- 22:38Data migration and deletion workflows
- 24:43Achieving goals and conclusion
related talks
20mDelta Lake: Turn your Data Lake into a Lakehouse
34mTales of Data Architecture Evolution
01h 03mReinvente su estrategia de datos con Lagos de Datos Transaccionales
21mBuilding Data Integration Services In Real Time
53mData Mesh in Practice - Assuring Data Quality at Scale
58mBuild reliable data lakes with Delta Lake & Databricks
41mTransactional data lakes for your analytical workloads
39mDelta Lake: Diving into Data Lakes Without the Downsides