Moving Beyond Moving Bytes
summary
Claim: streaming pipelines must use shared schemas to enable projection, join and aggregation rather than “moving bytes.” Demo wires Apache NiFi → Kafka → Flink with Confluent and Hortonworks registries, showing end‑to‑end serialization, registry lookup and Flink deserialization working live. Registry features shown: centralized versioning and REST lookup, client caches with TTL, and header formats (Confluent: 5‑byte header, Hortonworks: 13‑byte header). Example proves registry reduces payload bloat vs embedded schemas and supports schema evolution via versioned entries.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:05What is a schema registry
- 5:20Schema registry landscape
- 9:49Schema registry wire formats
- 10:11Using schema registry portal
- 11:17Schema registry in data pipelines
- 14:06Data acquisition example
- 22:38Live demo walkthrough
- 28:14Future of schema registries
- 29:00Q&A session
related talks
38mStreaming with Structure
29mHigh-Quality Performant and Cost Efficient Schema-Aware Data Streams on Flink at Netflix Scale
40mThe Seven Deadly Sins of Streaming
25mFrom Guesswork to Governance: Bringing API Discipline to Data Streaming
32mKeynote: Apache Flink - Completing Cloudera’s End to End Streaming Platform
31mEnd-to-end Flink SQL with Schema Registry, Kafka & Kudu
31mOne Data Pipeline to Rule Them All
39mUsing Streaming Data Pipelines in the Cloud to Achieve Success