Advancing Data Mesh: Building A Stream Processing EcoSystem of Reusable Processors and Datasets
summary
Netflix builds a reusable stream platform that runs in production and is GA as of Oct 2021, powering 160+ pipelines in production and using Flink + Kafka under the hood. It standardizes sources as key‑compacted topics, offers a processor SDK and a drag‑and‑drop UI for no‑code pipelines, enforces schema‑first contracts with 4,000+ schemas registered and an Iceberg schema with ID‑based mapping, and automates per‑consumer schema evolution and safe deployments.
AI-written summary · about
chapters (4)
related talks
34mCustom, Complex Windows at Scale using Apache Flink
43mStructure and Interpretation of Stream Processing
38mBuilding Unified Streaming Platform at Uber
31mWriting an interactive streaming SQL engine and pre-parser using Flink
28mPlaying to Win: Why Netflix Platform Engineering Chose Fed GraphQL
31mThe mesh inside the mesh
37mKeynote | Flink First: Designing Modern Trading Systems
25mMixing data streams at scale