Metadata Automation in the Data Mesh
summary
Zalando faces 50 million customers, ~150 data scientists, 500 AWS accounts and thousands of services and builds a unified metadata platform plus a data product toolbox that wires Databricks Unity Catalog, Airflow, Glue and custom quality tooling into discovery and governance. A metric definition layer propagates semantics across hundreds of tables. Retention gets enforced and unused datasets get pruned, which saved 100,000 euro per month. Federated metadata distribution works in production and generative AI for discoverability is next.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 5:05Zalando’s data strategy
- 9:29What is data mesh
- 10:36Four key ingredients
- 15:24Data product journey
- 17:10Metadata roles and types
- 23:18Building unified metadata
- 26:58Metadata automation benefits
- 29:08Big data vs great data
- 30:54Lessons from automation
- 33:45User journey and platform
- 34:30Road ahead
related talks
54mData Mesh in Practice
24mMetadata driven framework for data discovery and data quality
46mJourney of metadata
39mData Mesh: Data Analytics for Engineers
40mData Intelligence Platform on AWS
40mData Mesh, Data Fabric and DataOps; what should you choose?
39mData Mesh in the wild
46mA Data Mesh Reference Implementation on AWS