The Incremental ETL Architecture

on YouTube ↗30m

" summary

Incremental ETL at big scale uses Apache Spark Structured Streaming with Delta Lake tables and Databricks Autoloader to move only new or changed rows, keeping history via table versioning and checkpoints. The architecture implements bronze/silver/gold medallion layers, uses a foreachBatch loop with merge into (upsert) to update a single changed row among batches, and supports switching between continuous streaming and scheduled runs via triggerOnce. It demonstrates reliable, low-latency updates without rewriting whole files.

AI-written summary · about

" chapters (12)

1. 0:00Introduction to incremental ETL
2. 1:42Why incremental ETL matters
3. 8:10Advantages of incremental ETL
4. 12:05Ingest in incremental ETL
5. 14:34Structured Streaming for ETL
6. 17:27Continuous vs scheduled processing
7. 19:12Delta Lake for updates and deletes
8. 21:01Streaming between Delta tables
9. 22:23Schema evolution in Delta
10. 24:26Example of incremental ETL
11. 27:36Upserting data in real time
12. 29:41Q&A and closing remarks

" related talks

• 50mIntroduction to Databricks Delta Live TablesNiall Langley · SQLBits 2023
• 22mReliable data engineering made easyMahdi Askari · DataEngBytes 2021
• 21mBuilding Data Integration Services In Real TimeKerry McRae · DataEngBytes 2020
• 45mSimplify your ETL with Databricks Delta LakePiotr Mucha · SQLBits 2020
• 51mDeclarative ETL pipelines with Delta Live TablesVuong Nguyen · SQLBits 2023
• 44mThe Changing Face of ETL: Event-Driven Architectures for Data EngineersRobin Moffatt · code.talks 2019
• 34mTales of Data Architecture EvolutionJosef Goldstein · Build Stuff 2022
• 04mInterviewMichael Paul Armbrust · Scale By The Bay 2019