Apache Kafka Meets Apache Iceberg: Real-Time Data Streaming
summary
Shows a working pipeline that materializes Kafka topics as Iceberg tables with a single click using Confluent Tableflow, taking a Postgres CDC stream through Postgres CDC to Kafka topics (products, orders), running Flink SQL to produce an aggregated product_sales topic, and exposing results via a built‑in Iceberg REST catalog so both Athena and Snowflake queries read the same Iceberg data. Demonstrates schema evolution, time travel, hidden partitioning, ongoing compaction and that queries return live streaming data without manual conversion.
AI-written summary · about
chapters (11)
- 0:00Intro
- 1:10Overview
- 2:06Kafka is the standard for operational data
- 3:41Iceberg for analytical data in data lakes
- 4:42Apache Iceberg
- 5:27Why Iceberg?
- 12:24Structure of an Iceberg table
- 16:40Streaming to data lakes is complicated
- 20:47Tableflow materialize Kafka topics as Iceberg tables
- 23:47Demo
- 35:37Outro
related talks
38mMelting Icebergs: Direct access to Kafka Data via Iceberg
39mApache Iceberg ingestion with Apache NiFi
28mStreaming Lakehouse with Kafka, Flink and Iceberg
42mTurning the database inside out again
16mLife is But a (Data) Stream: Building Quality Data Pipelines
25mHow to build a platform for AI and analytics based on Apache Iceberg
41mTip of the Iceberg
50mDesigning a scalable data platform based on Apache Spark/Iceberg