Platform Building for Data Mesh - Show me how it is done!
summary
A concrete platform is built that models each data product as a Kubernetes CRD and uses the operator pattern (Micronaut Java operators) to reconcile desired state, bootstrap a git repo and CI/CD, provision namespaces and S3 buckets, and wire outputs behind Istio URLs. Outputs target ClickHouse/BigQuery, data lakes, triple stores and streaming, while a DataHub catalog ingests metadata and Knative CloudEvents drive freshness across upstream/downstream products. The setup automates lifecycle and discovery so teams self-serve data products.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 0:26Agenda
- 0:57What is Data Mesh
- 4:27Guiding Principles
- 5:22Platform Components
- 8:06Data Mesh Platform
- 19:00Registration
- 23:06Deploy and Provision
- 29:19Data Discovery Catalog
- 30:54Eventing
- 33:24Summary
- 36:19Outro
related talks
34mTransforming Data Processing with Kubernetes: Journey Towards a Self-Serve Data Mesh
36mIntroducing Data Mesh A Paradigm Shift In Analytical Data Management Architecture
48mBuilding a data mesh
30mData Mesh – Architecture And Insights From The Self-service Platform Boosting Financial Services
30mPlatform Oriented Data Architecture for Acceler...
43mWhen design meets data: Using Data Mesh for a consumer and business-driven data strategy
31mData Mesh In Practice – How To Set Up A Data Driven Organization
43mUntangling Data Mesh: A paradigm shift in Big Data