Streaming Event-Time Partitioning With Apache Flink and Apache Iceberg
summary
Netflix builds a Flink router that writes playback session summaries into Apache Iceberg tables partitioned by event time, using Iceberg atomic commits tied to Flink checkpoints for exactly-once output. Deduplication state drops repeats, late events buffer and flush periodically, and a dynamic partitioner adjusts write parallelism per partition. A naive sink produced 250 million files per day; after batching, dynamic partitioning and background compaction this drops to about 100,000 files and has run in production for a quarter.
AI-written summary · about
chapters (14)
- 0:00Intro
- 0:19What is this talk about?
- 1:54Table Partitioning
- 4:32Before: Playback Data
- 6:21Common Pattern
- 7:15Better Pattern?
- 7:47Why is this hard?
- 9:00Iceberg Sink
- 10:21This should be easy...
- 15:21After: Playback Data
- 16:02Generic event-time partitioning out of the box
- 16:50New Pattern
- 17:06Tradeoffs
- 18:34And next time...
related talks
24mEfficient, Low Latency Ingestion to Large Tables via Apache Flink and Apache Iceberg
30mBackfill Flink Data Pipelines with Iceberg Connector
43mA Data Streaming Architecture with Apache Flink #bbuzz
42mStream Processing with Apache Flink
20mBuilding metric platform using Flink for massive scale at Netflix
39mApache Iceberg ingestion with Apache NiFi
34mBeaming Flink to the Cloud @ Netflix
28mStreaming Lakehouse with Kafka, Flink and Iceberg