Data Mesh in Practice - Assuring Data Quality at Scale
summary
Built a central, self-serve data quality platform that monitors thousands of streams and ingests tens of billions of events and hundreds of terabytes per day into S3, using Apache Flink for processing and Kafka/Hive for sources. It profiles data to compute completeness, accuracy, consistency and timeliness, runs threshold and anomaly checks, and integrates with Slack, email and PagerDuty plus a feedback loop to prune false alerts. The platform scales but anomaly detection and varied ingestion patterns require custom engineering.
AI-written summary · about
chapters (10)
- 0:00Introduction and context
- 3:00Data quality challenges
- 5:13Defining data quality
- 10:05Common data quality issues
- 14:26Measuring data quality
- 20:08Objective vs subjective metrics
- 24:38Data mesh and quality
- 31:12Building a central platform
- 38:32Platform implementation steps
- 46:52Lessons learned and conclusion
related talks
34mTransforming Data Processing with Kubernetes: Journey Towards a Self-Serve Data Mesh
57mOur DataMesh Journey: data architectures for microservices
01h 02mShift Data Quality Left with Data Mesh Principles
44mData Mesh
17mData Mesh: The Speedrun
48mData Mesh Paradigm Shift in Data Platform Architecture -
39mCrafting modern data platforms: reveal your data value
27mThe journey of Data Mesh and Compliance in a Multi-Regional Data Lake