Building metric platform using Flink for massive scale at Netflix
summary
A config-driven Flink platform produces sessionized metrics at Netflix scale handling ~260 billion events/day and terabytes of RocksDB state for 150+ million members, using a schema registry, UDF store and filter, sessionizer, summarizer components. Schema evolution lets teams add metrics without redeploys; switching to compact arrays and Avro binary serialization cut checkpoint state ~30% and stabilized jobs. The system uses injection-time watermarks to avoid noisy event clocks, runs as the largest stateful Flink job at Netflix, and reliably serves many downstream summarizers.
AI-written summary · about
chapters (10)
- 0:00Intro
- 0:35What type of metrics are required?
- 1:58What are different session definitions?
- 4:29Events traffic scale
- 5:06Users want those metrics fast!
- 6:11The Design & Inner workings
- 14:09Lessons learnt
- 15:45Ingestion time for watermark instead of event time
- 17:37Compact array representation reduces state size by 2x compared to Map
- 18:36Avro binary serialization further reduces the state size by 35%
related talks
34mBeaming Flink to the Cloud @ Netflix
43mLarge Scale User Behavior Analytics by Flink
47mBring Flink to Large Scale Production
19mStreaming Event-Time Partitioning With Apache Flink and Apache Iceberg
36mDynamically Generated Flink Jobs at Scale
45mApache Flink — Real-Time Data Processing
35mApache Flink 2.0: Unified Enterprise Data Processing System and Beyond
31mFlink @WORKDAY: scaling of ingestion data pipelines to S3