Apache Hudi: A Database Layer Over Cloud Storage for Fast Mutations & Queries
summary
Apache Hoodie implements a storage engine over cloud object storage with file groups with base + log files and a timeline stored as an LSM to support transactional commits, scaling to multi‑exabyte deployments used at Uber, Amazon and Walmart. It moves from 8‑hour bulk ingests to ingest every 20 minutes, offers copy‑on‑write and merge‑on‑read tables, hash/bloom/partition indexes, columnar delta writes beside Parquet/Avro base files, CDC + event‑time merge, incremental queries, and non‑blocking concurrency for concurrent writers.
AI-written summary · about
chapters (19)
- 0:00Apache Hudi intro
- 1:23Hudi origins
- 9:07Lakehouse motivations
- 14:26Hudi storage layout
- 18:36Copy-on-write vs merge-on-read
- 20:43Hudi timeline and logs
- 26:31Metadata management
- 30:26Storage engine trade-offs
- 34:21Scaling metadata
- 38:11Concurrency control
- 42:04Writer starvation and compaction
- 45:17Read and query performance
- 47:24Writer-side indexes
- 51:40Reader-side indexes
- 52:02Merge operations optimization
- 55:57Streaming in lakehouse
- 1:00:37Hudi deployment
- 1:02:19Storage vs table format
- 1:03:42Hudi future
related talks
50mApache Hudi : The Data Lake Platform
31mBuilding large scale, transactional data lakes with Apache Hudi
34mApache Iceberg.
23mHadoop & Cloud Storage: Object Store Integration in Production
31mPanel: Open Data Lakes
55mApache HBase: crazy dances on the elephant back
25mBuild open format data lakehouse
39mApache Druid 101