Our DataMesh Journey: data architectures for microservices
summary
Company runs 250+ microservices and 300+ developers with 60–100 releases per day and ~2 billion ML predictions/day, so data sharing moves from sync queries to event streaming. Implementation uses Kafka + Avro with one schema repo per event, a YAML registry to drive a domain-event mesh loader that writes one BigQuery table per event, automated cataloging and data-loss scans, and strict rule: publish only IDs and minimal owned fields. Result: lower producer coupling, consumer-pays compute, and operational governance by automation.
AI-written summary · about
chapters (36)
- 0:00Introduction
- 0:23Agenda
- 2:09Context
- 3:19The monolith
- 4:08Splitting the monolith
- 5:22Data sharing
- 6:26Simple query
- 7:03Good and bad things
- 8:18Booking indexer
- 9:16Public indexer
- 10:15Booking engine
- 11:04Query by ID
- 11:52Learnings
- 13:32Domain Events
- 16:19Implementation details
- 18:13Landing pages and bookings
- 19:30Analytical BI
- 22:41Data lakes
- 25:34DataMesh
- 28:07Architecture Design Implementation
- 29:18Budget
- 30:09Rollout
- 31:52Architecture
- 35:04Data Loss
- 35:32Data Loss Metrics
- 36:58Feedback Loops
- 39:19Questions
- 41:02How to document domain events
- 43:01Publishing events
- 44:37How do you decide which data goes
- 46:50Can I store this data
- 48:42How do I know the state of the group
- 51:14Single point of failure
- 51:51Snapshots
- 53:18Data persistence
- 54:58Main events
related talks
17mData Mesh: The Speedrun
34mTransforming Data Processing with Kubernetes: Journey Towards a Self-Serve Data Mesh
53mData Mesh in Practice - Assuring Data Quality at Scale
48mBuilding a data mesh
44mData Mesh
39mData Mesh: Why, What and How
20mEmbracing Data Mesh: A Modern Strategy for Scalable Data Management
31mData Mesh In Practice – How To Set Up A Data Driven Organization