Remove Slow, Expensive ETL & Bring Data To The Data Lake For Lower Cost, More Performant Analytics
summary
Claim: replace slow, costly ETL with instant CDC-fed tables in the lake to cut costs and speed analytics. Implementation uses Debezium -> Kafka -> Iceberg to stream operational DB changes into Parquet snapshots, adds Project Nessie for branching and history, runs transformations with dbt, and serves queries with Dremio. Result is ACID tables on object storage, schema evolution, ability to branch/test whole catalogs, and typical migrations shrinking from months to days or minutes.
AI-written summary · about
chapters (11)
- 0:00Introduction and agenda
- 1:49Evolution of data architectures
- 3:52From warehouses to lake houses
- 5:15Extracting data with Debezium
- 7:03Iceberg table format explained
- 9:13Metadata and catalog challenges
- 10:17Project Nessie for multi-table transactions
- 13:25Transforming data with DBT
- 16:36Querying the open lake house
- 20:11Benefits of interactive data lake engines
- 23:00Summary and key takeaways
related talks
11mBreaking Data Silos: Enterprise Data Management
01h 03mReinvente su estrategia de datos con Lagos de Datos Transaccionales
34mData Lakehouse: Who, Why and How
13mBuild your open data lakehouse with Dremio and AWS (PRT085)
39mApache Iceberg Merge-On-Read: Streaming CDC
45mModern Data Architecture
26mModernize your Enterprise Data Warehouse & generate value from data
58mBuild reliable data lakes with Delta Lake & Databricks