Take the Polar Plunge: A Fearless Introduction to Apache Iceberg™
summary
Introduces Apache Iceberg as an open table format that adds a metadata layer (manifest files, manifest lists, metadata file) on object storage to get ACID transactions, schema evolution and hidden partitioning while using engines like Apache Spark or Flink. Explains catalog role, shows a live Spark SQL + Apache Polaris notebook where creating a bookings table, changing partition spec from hour to day, inserting and time-traveling snapshots all succeed. Covers update strategies (copy-on-write vs merge-on-read), warns rewriting a 500 MB file to change one row, and recommends compaction and manifest rewrite.
AI-written summary · about
chapters (15)
- 0:00Introduction to Apache Iceberg
- 2:26Data warehouses limitations
- 6:01Data lakes and their issues
- 8:06Introducing data lakehouses
- 10:56Iceberg architecture overview
- 14:24Choosing a catalog for Iceberg
- 17:55Querying Iceberg tables basics
- 22:44Hidden partitioning explained
- 25:33Time travel in Iceberg
- 27:20Merge and upsert operations
- 28:26Copy-on-write vs merge-on-read
- 31:58Handling small files in Iceberg
- 34:47Compaction and maintenance
- 37:14Live demo and setup
- 43:01Q&A session
related talks
41mTip of the Iceberg
01h 12mHands-on with Apache Iceberg
01h 09mMore Than Just The Tip Of The Iceberg
31mHow Apache Iceberg Enables Multi-Engine Data Platforms
37mApache Iceberg: An Architectural Look Under the Covers
52mApache Iceberg - An Architectural Look Under the Covers
01h 04mAn Extremely Technical Overview of How Apache Iceberg Planning Actually Works
39mIntroducing Apache Iceberg: Tables Designed for Object Stores