An Extremely Technical Overview of How Apache Iceberg Planning Actually Works
summary
Detailed walkthrough of the Java implementation of Apache Iceberg planning shows how Spark SQL push predicates convert to iceberg expressions, how manifest-list and manifest partition-summary filtering use min/max and bucket transforms to skip files, and how a delete file index pairs equality/position/delete-vector files to data files before producing split-sized scan tasks. The code claims to reduce I/O by reading only relevant manifests and data-file metrics, and V4 plans typed column stats and collapsing manifest lists into unified manifests.
AI-written summary · about
chapters (17)
- 0:00Introduction
- 1:02Apache Iceberg planning benefits
- 5:16Iceberg architecture basics
- 8:03Table format design
- 10:52Snapshots and commits
- 13:39REST protocol future
- 15:01Snapshots schema overview
- 16:25Schema evolution with field IDs
- 22:51Hidden partitioning
- 26:45SparkSQL integration
- 34:49Manifest filtering
- 40:27File metrics evaluation
- 48:09Delete files handling
- 53:00Scan tasks building
- 54:00Iceberg V4 changes
- 56:28Iceberg community future
- 57:38Q&A
related talks
52mApache Iceberg - An Architectural Look Under the Covers
37mApache Iceberg: An Architectural Look Under the Covers
49mHow Apache Iceberg Really Works
46mTake the Polar Plunge: A Fearless Introduction to Apache Iceberg™
34mApache Iceberg.
41mTip of the Iceberg
01h 12mHands-on with Apache Iceberg
50mDesigning a scalable data platform based on Apache Spark/Iceberg