Build simple & scalable data pipelines with Polars & DeltaLake
summary
Claims that Python + delta-rs (Delta Lake Rust) plus Polars projection/predicate pushdown let pipelines handle evolving messy data with real transactions. Demo writes Parquet-backed Delta tables, runs a merge/upsert and time travel workflow that updates exactly two bad rows, compacts three small parquet files into one, supports schema evolution, and queries partitioned S3 data with file skipping. Delta RS runs without Spark/Java; compaction, merge, versioning and predicate pushdown actually work, with some S3 parallel-write caveats.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 1:02Live data pipeline challenges
- 4:53Solution with Polars & Delta Lake
- 8:27Demo: first Delta table
- 11:38Appending data to Delta
- 14:07Updating with merge
- 17:36Schema evolution & versioning
- 20:05Time travel & restore
- 21:09Compaction for small files
- 22:14Query optimizations
- 24:20Partitioned datasets in cloud
- 25:23Key takeaways
- 26:12Q&A
related talks
38mGetting Started with Delta Lake
01h 28mBuilding Reliable, Open Lakehouses with Delta Lake
26mBuilding Python Binding for the Delta Lake Library in Rust
19mOpen Source Reliability for Data Lakes
31mDelta Lake: Open Source Reliability and Quality for Data Lakes
42mBuilding a Lakehouse data platform using Delta Lake, PySpark, and Trino
46mOpen Table Formats in the Wild: From Parquet to Delta Lake and Back
31mSimplifying upserts and deletes on Delta Lake tables