Streaming Lakehouse with Kafka, Flink and Iceberg
summary
Claimed end-to-end streaming lakehouse that ingests synthetic pageclicks with Kafka Connect data-gen, enriches and joins streams with static files using Flink joins and writes to Iceberg, and stores Parquet data on MinIO with Tabular catalog, queried by Trino and visualized in Superset. Demo runs as a Docker Compose stack, shows real-time pie charts of top regions and pages, and highlights Iceberg features used: time travel, schema and partition evolution, compaction to combat small files.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:27Data warehouse challenges
- 4:59Lakehouse architecture
- 7:31Streaming lakehouse
- 9:17Lakehouse components
- 13:28Apache Iceberg deep dive
- 18:23Iceberg as open standard
- 20:33Kafka and Flink in lakehouse
- 21:57Lakehouse limitations
- 23:01Demo and GitHub
- 26:11Open data future
related talks
36mApache Kafka Meets Apache Iceberg: Real-Time Data Streaming
45mBuilding an Open Data Lakehouse using Apache Spark, Apache Iceberg and Dremio
31mFrom Warehouse to Lakehouse: Data at Rest
27mStream, Store, Visualize: Fast-Track Real-Time Analytics the OSS Way
26mArchitecting real-time transactional data lake
36mHail Hydrate! From Stream to Lake
38mMelting Icebergs: Direct access to Kafka Data via Iceberg
29mLakeside AI: Bringing Models to Data with Open Lakehouses & Agents)