Truly Open Lineage
summary
Three LF AI & Data projects integrate to record both deployed structure and actual runs: Egeria captures design metadata, OpenLineage defines an HTTP “Run” event with extensible facets, and Marquez is the reference event store and visualizer. A small proxy backend publishes Run events from Spark/DBT/pandas into Kafka from a secure enclave so operational lineage (run counts, row totals, timings) stitches to design lineage; the pipeline works end-to-end and flags missing or delayed runs.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:07Data lineage importance
- 5:39Traditional lineage limits
- 8:27Design vs operational lineage
- 13:23Open Lineage standard
- 15:31Open Lineage event structure
- 18:24Proxy backend in practice
- 23:02Marquez reference implementation
- 23:46Collaboration benefits
- 28:08Get involved
- 31:20Q&A
related talks
46mCross-Platform Data Lineage with OpenLineage
30mCross-Platform Data Lineage with OpenLineage, A Foundational...
41mObservability for data pipelines with OpenLineage
34mColumn-level lineage is coming to the rescue
35mAutomating Airflow Backfills with Marquez
45mData Lineage with Apache Airflow using Marquez
16mArchitecting Scalable Data Lineage Graph for Privacy Compliance and Agentic Analysis
28mthe semantic layer your AI needs