Let's Do Data Lineage in Kafka, Flink and Druid by Tracking Aircraft!
summary
Shows an end-to-end KFD pipeline that ingests real ADS-B aircraft feeds from Raspberry Pi receivers into Kafka, enriches and decodes records with Flink, and lands them in Apachi Druid while preserving per-record lineage in Kafka headers. Demonstrates headers surviving Flink transforms and appearing as metadata columns in Druid so queries can trace each record back to a receiver, using a mix of cloud Kafka and local Docker for the demo and live ADS-B feed aggregation.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 1:54Data governance and lineage
- 7:16Lineage with Kafka headers
- 15:12Why lineage matters
- 18:44Aircraft data with Raspberry Pi
- 27:14Loading data into Druid
- 32:56End-to-end lineage tracking
- 33:39Key takeaways
- 35:06Q&A
related talks
20mReal-time analytics with Flink and Druid #bbuzz
37mBuilding an Event Analytics Pipeline with Kafka, ksqlDB, and Druid
41mBuilding an Open Source Streaming Analytics Stack with Kafka and Druid
27mSucceeding with Apache Druid and Clickstream Data
49mLet’s Build a Streaming Data Pipeline!
20mAnalyzing streaming data with Apache Druid
27mLet Your Data Tell Its Story
46mScalable Realtime Analytics using Druid