Cross-Platform Data Lineage with OpenLineage, A Foundational...
summary
Shows a concrete JSON‑schema standard with Java and Python libs and active integrations for Airflow, Spark, dbt, Flink, Trino and Snowflake; emits run events (start/complete/running) tying job runs to inputs and outputs so an upstream integration bug that inverted source values can be traced from dataset to ML model to app. Implements schema facet and parent run facet for column‑level diffs and DAG hierarchies, has a reference consumer (Marquez) and is already used by Google, AWS and Datadog.
AI-written summary · about
chapters (12)
- 0:00Data observability with OpenLineage
- 2:46Limitations of observability tools
- 4:35Data lineage importance
- 7:49OpenLineage as vendor-neutral standard
- 11:06OpenLineage core concepts
- 16:01OpenLineage event modeling
- 18:48OpenLineage in multi-tool pipelines
- 21:16Standard and custom facets
- 22:23OpenLineage ecosystem
- 25:13OpenLineage at Datafold
- 28:50Cross-platform lineage benefits
- 29:54Q&A
related talks
46mCross-Platform Data Lineage with OpenLineage
41mObservability for data pipelines with OpenLineage
32mTruly Open Lineage
34mColumn-level lineage is coming to the rescue
45mData Lineage with Apache Airflow using Marquez
35mHow to build Data Lineage on Apache Spark logs
35mAutomating Airflow Backfills with Marquez
28mBuilding A Thriving Data Culture That Ignites The Fire To Fuel Innovation