Let’s see how fast Impala runs on Iceberg
summary
Impala uses a native C++ Iceberg reader and new Iceberg-specific operators to apply positional delete files; a directed distribution mode maps delete-file paths to the nodes that actually read those data files, cutting exchanged rows from ~9.5B to 825M and lowering a simple count query from ~21s to ~12s (≈42% faster). On a 1T-row grouped benchmark with ~70B positional deletes a naïve run rises from 2.5m to 7m15s; build-side batching and parallel sort reduce that to ~5m. Future plans include Roaring Bitmap for compact deletes; TPC-DS 1k compares Impala (≈15.5m) vs Engine X (>20m, ~1.3×).
AI-written summary · about
chapters (16)
- 0:00Introduction and presenters
- 1:14Impala architecture overview
- 4:25Iceberg delete strategies
- 8:15Performance test setup
- 9:21Network transfer bottleneck
- 10:45Directed distribution mode
- 13:35Zan takes over presentation
- 13:561 trillion row challenge
- 15:23Iceberg V2 with deletes
- 16:53Performance impact of deletes
- 17:35Join operator details
- 20:44Optimizing join build operator
- 22:31Future optimizations
- 23:54Exchange operator inefficiency
- 24:37Comparison with Engine X
- 26:21Q&A and closing
related talks
42mYou Migrated to Apache Iceberg—Now What? Speed, Efficiency, and Cost in the Real World
05mMaking Iceberg walk and talk like Postgres
52mApache Iceberg - An Architectural Look Under the Covers
41mTip of the Iceberg
01h 12mHands-on with Apache Iceberg
37mApache Iceberg: An Architectural Look Under the Covers
34mApache Iceberg.
01h 04mAn Extremely Technical Overview of How Apache Iceberg Planning Actually Works