Gatekeep iceberg data quality with Apache Toree and airflow
summary
Uses Apache Iceberg column-level stats (min/max, null fraction, distinct counts) to run deterministic data-quality rules and a write‑audit‑publish (WAP) gate that keeps new snapshots non‑visible until checks pass; explores and prototypes checks interactively in Jupyter via Apache Toree (Scala/Spark kernel) and converts notebooks into automated Airflow DAGs with sensors/operators and a notebook operator. Demonstrates on a 684,000-tree NYC dataset, shows publishing is a metadata‑only snapshot flip (no file rewrites) so pipelines can pause safely until validation succeeds.
AI-written summary · about
chapters (16)
- 0:00Introduction to data quality
- 2:23Why data quality matters
- 4:10NASA data quality failure case
- 6:17Data quality checks and Iceberg
- 9:11Write audit publish pattern
- 10:36New York tree census example
- 13:46Data quality decisions
- 14:50Iceberg snapshot handling
- 15:35Iceberg branching and tagging
- 16:38Interactive data exploration
- 17:43Apache Toree and Jupyter
- 18:47Airflow for automation
- 19:30Key data quality checks
- 20:14Notebook-based operators
- 20:58Airflow workflow integration
- 22:02Data lineage importance
related talks
50mDesigning a scalable data platform based on Apache Spark/Iceberg
41mTip of the Iceberg
39mApache Iceberg ingestion with Apache NiFi
01h 09mMore Than Just The Tip Of The Iceberg
39mIntroducing Apache Iceberg: Tables Designed for Object Stores
52mApache Iceberg - An Architectural Look Under the Covers
43mFloe: Policy-Based Table Maintenance for Apache Iceberg
36mApache Kafka Meets Apache Iceberg: Real-Time Data Streaming