Backfill Flink Data Pipelines with Iceberg Connector
summary
Netflix builds an Iceberg source connector on Flink Flip 27 source API that reads Parquet-backed Iceberg tables while mirroring Kafka-like ordering via a global injection watermark, split enumerator and readers. Backfills 24 hours of impressions in ~5 hours, yields 99.9% match with production, and lowers retention cost from about $93M/yr in Kafka to ~$2M/yr in Iceberg. Supports configurable lateness (example 15 minutes), requires ~8 hours of data for RMI backfills, and handles multi-source alignment.
AI-written summary · about
chapters (5)
- 0:00Needs for backfilling Flink Applications
- 2:43Existing approaches
- 7:04Iceberg Source
- 13:26Event ordering challenges
- 25:11Enabling Iceberg backfill
related talks
19mStreaming Event-Time Partitioning With Apache Flink and Apache Iceberg
36mApache Kafka Meets Apache Iceberg: Real-Time Data Streaming
39mApache Iceberg ingestion with Apache NiFi
38mMelting Icebergs: Direct access to Kafka Data via Iceberg
24mEfficient, Low Latency Ingestion to Large Tables via Apache Flink and Apache Iceberg
28mStreaming Lakehouse with Kafka, Flink and Iceberg
31mHow Apache Iceberg Enables Multi-Engine Data Platforms
42mTurning the database inside out again