The Who, What, and Why of Data Lake Table Formats
summary
Compares Apache Iceberg, Apache Hudi, Delta Lake and states concrete tradeoffs: Hudi prioritizes ultra‑fast ingestion and transactional updates (Uber‑style workloads), Iceberg emphasizes batch, time‑travel snapshots, hidden partitioning and schema evolution for reproducible ML, and Delta focuses on streaming inside the Databricks ecosystem. Claims ACID transactions, metadata catalogs and S3 support make table formats usable on a lakehouse; notes recent vendor moves (Databricks/Unity and S3 first‑class Iceberg support) that push Iceberg toward de facto standard.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:08Data lakehouse overview
- 3:55Database evolution
- 7:26Data lakes and challenges
- 10:32Lakehouse concept
- 12:21Table formats for lakehouse
- 13:25Apache Hudi use cases
- 15:15Apache Iceberg benefits
- 17:01Delta Lake streaming
- 18:26Why formats now
- 20:12Future of formats
- 21:15Q&A
related talks
25mBuild open format data lakehouse
25mDelta Lake has competition
46mOpen Table Formats in the Wild: From Parquet to Delta Lake and Back
34mData Lakehouse: Who, Why and How
51mSolving the challenges of adding transactional support to Data Lakes
41mTransactional data lakes for your analytical workloads
31mData Warehouses Meet Data Lakes
49mHow Apache Iceberg Really Works