Firebolt: Why Powering User Facing Applications on Iceberg is Hard
summary
Firebolt adds Iceberg support but finds Iceberg’s hierarchical metadata and object storage latency create a ~0.5s baseline for cold queries, so Firebolt caches table snapshots in memory with a user-configurable staleness and refreshes them async to avoid S3 hops. It exposes a SQL TVF to list snapshot files for cardinality (uses Iceberg row counts but limited other stats), implements fire cache fingerprints to reuse hash joins (about 10x reuse in early metrics), uses coroutine-based Parquet readers with a sparse buffer manager reading 2 MB buffer blocks, row-group pruning (no page pruning), and bucket-aware file assignment to avoid shuffles; on a 1 TB TPC-like dataset a complex join scans 18 GB and runs in ~8.5s while many variants return sub-second via cache reuse.
AI-written summary · about
chapters (11)
- 0:00Introduction and context
- 1:47Apache Iceberg 101
- 4:58Challenges of low latency
- 11:13Metadata caching strategy
- 13:11Query planning optimization
- 20:50Partition-aware execution
- 26:14Data pruning techniques
- 28:55Sub-result and result reuse
- 37:59Scalable scan architecture
- 45:08High-performance Parquet reading
- 57:01Demo and final discussion
related talks
01h 04mHow We Build Firebolt
15m100x faster data app experiences delivered cost-effectively
42mYou Migrated to Apache Iceberg—Now What? Speed, Efficiency, and Cost in the Real World
37mApache Iceberg: An Architectural Look Under the Covers
52mApache Iceberg - An Architectural Look Under the Covers
34mApache Iceberg.
41mTip of the Iceberg
05mMaking Iceberg walk and talk like Postgres