Moving Data From Graph to the Warehouse, A Configuration Driven Approach
summary
Netflix builds a configuration-driven pipeline that streams federated GraphQL types into the warehouse by listening to CDC events, enriching them via a stateless GraphQL processor, and writing shaped tables to Apache Iceberg (also supporting Hollow). YAML configs live in Git and lifecycle is handled by a Gradle plugin that registers versions in a data product registry, preserving ownership, schema, lineage and SLA. Works in real time but struggles with deeply nested arrays and differing Spark vs Trino reads.
AI-written summary · about
chapters (10)
- 0:00The value of federation
- 1:28The data warehouse struggle
- 2:31The data product pipeline
- 3:49Treating data as a product
- 5:28Architecting the pipeline
- 7:17Pipeline components defined
- 10:29Developer workflow integration
- 13:07Data product configuration
- 16:01Entity enrichment internals
- 19:44Scaling and future challenges
related talks
31mFrom Warehouse to Lakehouse: Data at Rest
31mOne Data Pipeline to Rule Them All
22mNetflix Data Mesh: Composable Data Processing
25mScaling data from the lake to the mesh via OSS
39mUsing Streaming Data Pipelines in the Cloud to Achieve Success
34mTransforming Data Processing with Kubernetes: Journey Towards a Self-Serve Data Mesh
45mModern Data Architecture
21mMigrate Data, Mesh in mind