Automating Airflow Backfills with Marquez
Talkat Open AI + Data Forum @ Open Source Summit North America 2022 · on Tuesday, 2022-06-21· at 11:10· Room 408/409 (Level 4), JW Marriott Austin, 110 E 2nd Street Austin, TX 78701·
summary
Shows how to use Airflow + Marquez + OpenLineage to automate downstream backfills by querying Marquez’s lineage REST API, traversing out-edges and invoking Airflow backfill for each dependent job. Describes a push-based OpenLineage integration, a Rust SQL parser for dataset extraction, a recursive Python client script that calls Airflow backfill, and a UI that maps job/dataset versions. Claims this runs in production and materially reduces manual cross-team debugging.
AI-written summary · about
chapters (12)
- 0:00Start
- 5:05Data Quality
- 6:34Upstream Failures
- 9:00Data Outages
- 9:34The Code Base Detection
- 9:44Manual Testing
- 10:41Execution Date
- 16:25What Effects if any Would Upstream Dax Have on Downstream Dags
- 22:40Integrations
- 24:40Operator Extractor
- 31:40Coordinate Efforts
- 33:08Job Hierarchy and Grouping
related talks
45mData Lineage with Apache Airflow using Marquez
46mCross-Platform Data Lineage with OpenLineage
30mCross-Platform Data Lineage with OpenLineage, A Foundational...
32mTruly Open Lineage
41mObservability for data pipelines with OpenLineage
34mColumn-level lineage is coming to the rescue
56mWhy Airflow is a secret ingredient in MLOps
26mGo with the Flow: Automating Your Workflows with Airflow