You Migrated to Apache Iceberg—Now What? Speed, Efficiency, and Cost in the Real World
summary
Catalog choice, query engine, file format and maintenance decisions determine whether Iceberg actually saves money and speeds queries: Avro writes 2–2.5× faster but produces tables that are 65% slower on reads and can scan ~5× more data unless compacted to Parquet; sorting cuts scanned data by ~51% in a TPCH 3 TB test and compresses data ~10% (sometimes 40%); Puffin files make TPCH twice as fast and individual queries 4–8× faster; a Rust compactor runs binpack in 190s vs Spark 1600s and reclaimed 112 TB of orphan files (~$2.5k/month).
AI-written summary · about
chapters (17)
- 0:00Introduction
- 2:54Apache Iceberg standard
- 5:25Legacy vs modern lakehouse
- 6:27Five Iceberg migration decisions
- 6:49Iceberg data catalog choice
- 10:04Query engines and Iceberg
- 14:13Iceberg data structure strategies
- 14:35Iceberg write strategies
- 16:41Avro vs Parquet performance
- 19:07Sorted vs unsorted data
- 22:40Delete formats trade-offs
- 25:28Hive to Iceberg migration
- 27:12Iceberg table optimizations
- 28:58Rust compaction engine
- 31:27Puffin files and acceleration
- 33:37Iceberg optimization summary
- 34:40Q&A
related talks
37mApache Iceberg: An Architectural Look Under the Covers
52mApache Iceberg - An Architectural Look Under the Covers
34mApache Iceberg.
41mTip of the Iceberg
46mTake the Polar Plunge: A Fearless Introduction to Apache Iceberg™
01h 08mFirebolt: Why Powering User Facing Applications on Iceberg is Hard
01h 04mAn Extremely Technical Overview of How Apache Iceberg Planning Actually Works
01h 12mHands-on with Apache Iceberg