Journey of metadata
summary
Describes a metadata platform that grows from a crawl-based “warehouse” to a distributed, model-first system powering over a million data assets and serving thousands of engineers and hundreds of teams; integrates Kafka, Hadoop/Hive, Elasticsearch, Neo4j (and LinkedIn’s Pegasus/espresso tooling), uses Calcite for fine-grained lineage, and emits ~3 million relationship events/day. Implements a generalized metadata architecture with anchors and aspects, provides REST and Kafka ingestion, surfaces via Data Hub, and proves it scales and supports compliance and analytics.
AI-written summary · about
chapters (27)
- 0:00Intro
- 1:24Linkedin's Data Ecosystem
- 4:44Crawl Phase
- 6:45Attempt 1: A few things we observed
- 9:34Walk Phase
- 12:55Attempt 2: A few things we observed
- 15:51What is Metadata?
- 17:33Some observations about the problem
- 18:48Run Together
- 20:12An Example Metadata Graph
- 21:36Metadata Serving
- 22:53Metadata Indexing
- 23:26Putting it all together
- 25:43Design Decision Cheat-Sheet
- 29:08What about X?
- 31:10Metadata Platform by the numbers
- 32:10A Web App: Data Hub
- 33:12Data Hub: Search
- 33:42Data Hub: Browse
- 34:16Data Hub: Lineage
- 34:33Data Hub: a Metric's page
- 34:51Data Management with Compliance
- 36:53Data Pipeline Operations
- 37:18Powered by Metadata
- 38:34Open Source : the details!
- 40:09Open sourcing the Data Model
- 40:34Open Source : what's coming next!
related talks
24mMetadata driven framework for data discovery and data quality
33mDiscover How to Create Your Own Metadata-Driven ML Platform from Scratch
29mMetadata Systems for End-to-End Data & Machine Learning
37mMetadata Automation in the Data Mesh
15mEasy Metadata Management (Git for Data) Using Neo4j
37mThe perils of building a democratic data platform
29mThe Data Journey
37mMetaOps: Metadata Operations For End-To-End Data & Machine Learning Platforms