Melting Icebergs: Direct access to Kafka Data via Iceberg
summary
A runtime projection presents Kafka topics as Iceberg tables without moving data by batching Avro messages into offset-range “files,” building Iceberg metadata on demand, and serving S3-compatible endpoints. It scans 24 hourly partitions for a one-day filter and 112 files for an unfiltered run in the demo, supports optional indexer-driven pruning for sub-second targeting, requires aligned hashing for bucket transforms, and remains a read-only projection with zero ETL and ephemeral Iceberg metadata. Performance depends on data shape and indexing.
AI-written summary · about
chapters (7)
- 0:00Introduction to Melting Icebergs
- 5:22Why Kafka and Iceberg now
- 11:23Iceberg overview and demo setup
- 24:41How the Kafka-Iceberg integration works
- 27:09Benefits of unified batch and real-time access
- 31:23Beyond Iceberg: Future logical projections
- 33:35Q&A session
related talks
36mApache Kafka Meets Apache Iceberg: Real-Time Data Streaming
41mTip of the Iceberg
42mTurning the database inside out again
39mIntroducing Apache Iceberg: Tables Designed for Object Stores
31mHow Apache Iceberg Enables Multi-Engine Data Platforms
34mApache Iceberg.
28mStreaming Lakehouse with Kafka, Flink and Iceberg
52mApache Iceberg - An Architectural Look Under the Covers