Building Reliable, Open Lakehouses with Delta Lake
summary
Shows a reproducible Docker demo environment and runs code that writes parquet files into a Delta table, updates the Delta transaction log and demonstrates atomic versioning, time travel, checkpoints and vacuum. Uses Delta RS and PySpark APIs, explains partitioning and Z-ordering for file skipping, and notes S3 put-object-atomicy races (~4 in 10 million) with a DynamoDB log-store workaround. Demonstrations succeed and illustrate how metadata + object storage yields ACID-like behavior.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 1:30Lakehouse motivation
- 3:16Setting up the Docker environment
- 11:01Data warehouse history
- 17:09Delta Lake intro
- 26:09Delta Lake APIs
- 30:58Regulatory data deletion
- 40:19Query performance tuning
- 47:47Docker build & notebook
- 51:59Creating Delta tables
- 1:01:26Overwriting data & logs
- 1:12:32Time travel & history
- 1:19:12Vacuum & retention
- 1:27:42Summary
related talks
38mGetting Started with Delta Lake
42mBuilding a Lakehouse data platform using Delta Lake, PySpark, and Trino
28mLakehouse Architecture with Delta Lake and Databricks
31mDelta Lake: Open Source Reliability and Quality for Data Lakes
18mDelta Lake – The Foundation Of Your Lakehouse
46mOpen Table Formats in the Wild: From Parquet to Delta Lake and Back
45mBuilding a Lakehouse on the Microsoft Intelligent Data Platform
19mOpen Source Reliability for Data Lakes