Let's Do Data Lineage in Kafka, Flink and Druid!
summary
Demonstrates a working pipeline that collects ADS‑B aircraft messages from Raspberry Pi ADS‑B feeders, sends them into Kafka with Kafka headers carrying provenance, decodes and normalizes semi‑structured payloads in Flink (using a Flink metadata column for headers), and ingests streamed rows into Druid for fast querying and dashboards. Shows headers survive processing, lineage appears as queryable columns, and live demos and dashboards confirm the approach end‑to‑end.
AI-written summary · about
chapters (9)
- 0:00Introduction to data lineage
- 3:15Data governance basics
- 7:18Why Druid for streaming
- 12:39Demo setup with ADS-B
- 15:11Loading data in Flink
- 19:50Streaming ingestion in Druid
- 23:28Visualizing lineage with dashboards
- 27:21Establishing chain of custody
- 28:46Resources and closing
related talks
47mBuilding Modern Data Streaming Apps with Open Source
27mSucceeding with Apache Druid and Clickstream Data
39mAnalysing Public Kafka Data from NASA Satellites
43mContinuous SQL with Kafka and Flink
36mApache Kafka Meets Apache Iceberg: Real-Time Data Streaming
03h 00mStream Processing, Going Beyond Pub/Sub with Kafka Streams and...
46mCross-Platform Data Lineage with OpenLineage
31mOne Data Pipeline to Rule Them All