Apache Iceberg Merge-On-Read: Streaming CDC
summary
Built a streaming CDC pipeline that tails MySQL binlogs with Debezium, publishes events to Kafka, consumes them with Shopify’s Pepto and writes Parquet files registered as Apache Iceberg tables using Iceberg v2 merge‑on‑read (equality and positional delete files). Ingestion lands in under seven minutes, compacted snapshot production ranges 3–40 minutes (biggest table ~40m) and steady‑state can approach ~7m; equality deletes and lots of small files require periodic compaction and tuned partitioning, max file group size and worker sizing.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 0:29Shopify data lake house
- 1:50Data ingestion challenges
- 3:41Legacy system limitations
- 6:51Snapshot production costs
- 8:26New system objectives
- 9:20CDC and Kafka ingestion
- 11:55Apache Iceberg overview
- 14:39Merge on read with Iceberg
- 18:38Performance results
- 21:36Lessons learned and closing
- 24:22Q&A session
related talks
25mRemove Slow, Expensive ETL & Bring Data To The Data Lake For Lower Cost, More Performant Analytics
36mApache Kafka Meets Apache Iceberg: Real-Time Data Streaming
39mIntroducing Apache Iceberg: Tables Designed for Object Stores
31mHow Apache Iceberg Enables Multi-Engine Data Platforms
34mApache Iceberg.
13mExploring the Depths of Apache Iceberg's Metadata Capabilities
26mArchitecting real-time transactional data lake
39mApache Iceberg ingestion with Apache NiFi