Open Table Formats in the Wild: From Parquet to Delta Lake and Back
summary
Attempts to replace a Parquet bronze layer with Delta Lake hit three concrete surprises: a 1.3 TB compaction test shows liquid clustering cuts API calls and latency versus Hive-style partitions, but a 50 GB incremental ingestion flips results—liquid clustering becomes slower and requires an undocumented Databricks config to behave; interoperability tests reveal Delta Lake feature incompatibility with DuckDB, Polars and Pandas until table features are stripped; file-based session boundaries only work with Autoloader, plain Delta tables break session semantics.
AI-written summary · about
chapters (14)
- 0:00Open table formats intro
- 1:55Why data matters
- 4:21Printing machine efficiency
- 7:29Big data platform evolution
- 12:46Parquet vs CSV/JSON
- 18:45Parquet limitations
- 20:28Delta Lake/Iceberg extend Parquet
- 25:25Testing Delta Lake
- 28:56Liquid clustering issues
- 31:26Delta Lake interoperability
- 33:52Session boundaries in Delta Lake
- 34:14Delta vs Iceberg communities
- 37:46Vendor lock-in reduction
- 38:50Q&A
related talks
39mDelta Lake: Diving into Data Lakes Without the Downsides
20mDelta Lake: Turn your Data Lake into a Lakehouse
28mLakehouse Architecture with Delta Lake and Databricks
58mBuild reliable data lakes with Delta Lake & Databricks
36mTable Formats Change Everything (By Not Changing Anything)
38mGetting Started with Delta Lake
19mOpen Source Reliability for Data Lakes
16mThe Evolution of Delta Lake