Delta Lake: Recompute without Bangs or Fuss
summary
A Spark job computes month‑to‑date revenue growth and writes results into a Delta Lake table using Dynamic Partition Overwrite and replaceWhere so only affected partitions or rows are updated. Airflow runs the single Spark program with start/end date parameters for per‑date replays, one‑click reruns, and a two‑year historical backfill; example runs show values changing (e.g. €50 → €500 → €56,000) and a backfill adjusting September to €46,000, illustrating the pipeline avoids heavy joins or full table overwrites.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 2:52Concrete use case
- 4:16Key technologies: Delta and Airflow
- 7:09Historical recovery and recomputations
- 9:37Update methods with Delta
- 12:07Handling incidents with Airflow
- 13:51Delta Lake benefits
- 14:33Delta vs Iceberg vs Databricks
- 15:19Q&A
related talks
31mDelta Lake: Open Source Reliability and Quality for Data Lakes
21mBuilding Data Integration Services In Real Time
45mSimplify your ETL with Databricks Delta Lake
39mDelta Lake: Diving into Data Lakes Without the Downsides
20mDelta Lake: Turn your Data Lake into a Lakehouse
31mUsing Spark to accelerate your lakehouse architecture with Fabric
22mReliable data engineering made easy
31mSimplifying upserts and deletes on Delta Lake tables