High-Quality Performant and Cost Efficient Schema-Aware Data Streams on Flink at Netflix Scale
summary
Netflix scales schema enforcement into Flink to generate and validate Avro records server-side for over 4 million events/sec and petabytes/day, using a centralized schema registry with local caching, asynchronous updates, and routing of non‑compliant JSON to a separate Kafka topic. The pipeline replaces JSON CPU work with Avro serialization, stores state as binary via RocksDB state in Avro, writes hybrid Iceberg tables, cuts compute ~30%, storage ~40% and achieves 40% cost savings.
AI-written summary · about
chapters (19)
- 0:00Intro
- 1:29The 3Vs of Big Data
- 3:32Data Variety at Netflix
- 5:10Data Variety Impact on Data Quality
- 6:03How to Handle Data Variety ?
- 6:50Use Case: Event Processing Pipeline
- 8:39Phases for Schema-aware Data Stream Design
- 9:13Design Challenges
- 11:14Schematization Design
- 13:59Design Approaches - Schemafication
- 14:53Schemafication Design
- 16:27Design Approaches - Schema Validation
- 17:50Schema Validation Sequence Diagram
- 20:02Schema Aware Data Streams Design
- 21:58Design Approaches - Data At Rest
- 23:00Data At Rest Design
- 24:22Schema-aware Data Streams Benefits
- 26:11An example of Compute Benefit
- 28:47Questions
related talks
34mBeaming Flink to the Cloud @ Netflix
39mMoving Beyond Moving Bytes
31mOne Data Pipeline to Rule Them All
39mUsing Streaming Data Pipelines in the Cloud to Achieve Success
31mEnd-to-end Flink SQL with Schema Registry, Kafka & Kudu
25mFrom Guesswork to Governance: Bringing API Discipline to Data Streaming
43mA Data Streaming Architecture with Apache Flink #bbuzz
26mData governance in streaming at scale