Column-level lineage is coming to the rescue
summary
OpenLineage adds a column-level lineage facet and demonstrates end-to-end extraction: a Spark listener inspects Spark logical plans at runtime and emits column mappings, a Rust SQL parser resolves SELECT and parses queries for column dependencies, and events stream to a Marquez backend* for visualization and API queries. Demos show automatic creation of dataset nodes and precise input→output column links across Spark, Airflow SQL operators, Postgres and BigQuery; the integrations work in real runs and resolve schema lookups at runtime.
AI-written summary · about
chapters (9)
- 0:00Intro
- 1:02Data Mesh
- 10:10Demo
- 14:00Data Engineering
- 16:27Spark Logical Plan
- 18:46Problems with Spark Logical Plan
- 23:46Future plans
- 24:31Summary
- 27:36Questions
related talks
30mCross-Platform Data Lineage with OpenLineage, A Foundational...
46mCross-Platform Data Lineage with OpenLineage
32mTruly Open Lineage
41mObservability for data pipelines with OpenLineage
35mHow to build Data Lineage on Apache Spark logs
16mArchitecting Scalable Data Lineage Graph for Privacy Compliance and Agentic Analysis
45mData Lineage with Apache Airflow using Marquez
35mAutomating Airflow Backfills with Marquez