Netflix Data Mesh: Composable Data Processing
summary
Netflix extends Keystone into a real-time Data Mesh that moves petabytes a day and trillions of events across systems for 150M members, built on Apache Flink jobs that read and write Kafka streams with a standardized Avro + Iceberg intermediate format. Pipelines compose reusable source, processor and sync connectors, deploy one Flink instance per pipeline, and inject auditors; schema evolution uses ID-based mapping and controller-driven redeploys to enforce fixed job schemas while enabling gradual, compatible changes.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 0:55What is Netflix
- 1:49Netflix is the Airport
- 9:18Data Mesh Platform
- 13:24Scheme Evolution
- 15:17Schema Changes
- 16:18Convergence
- 16:32Schema Compatibility
- 17:05ID Mapping
- 18:52Rename
- 19:44Physical S3
- 21:51Next Steps
related talks
34mBeaming Flink to the Cloud @ Netflix
34mTransforming Data Processing with Kubernetes: Journey Towards a Self-Serve Data Mesh
47mBuilding Modern Data Streaming Apps with Open Source
49mData Platform Architecture, Evolution, and Philosophy
53mData Mesh in Practice - Assuring Data Quality at Scale
25mMoving Data From Graph to the Warehouse, A Configuration Driven Approach
43mA Data Streaming Architecture with Apache Flink #bbuzz
57mOur DataMesh Journey: data architectures for microservices