Let Your Data Tell Its Story
summary
Claims a working, low‑overhead in‑house data‑lineage pipeline that uses Airflow Python Redis, sends deduplicated messages from central entry points, parses queries to emit per-column messages, and stores interactions in a lineage table. A separate worker splits messages, updates the table, and a recursive helper builds N‑level dependency graphs which feed pi viz visualization. It surfaces downstream consumers, true data origins, orphaned fields, and speeds root‑cause and impact analysis — built without a vendor.
AI-written summary · about
chapters (18)
- 0:00Introduction
- 0:29One data point's hidden story
- 3:21Why data lineage matters
- 4:03Planning data lineage
- 5:50Data interactions in pipelines
- 8:41Read/write interactions
- 10:51Functional requirements
- 12:14Solution architecture
- 13:17Extracting data details
- 15:43Breaking messages into points
- 17:51Populating lineage table
- 18:54Querying dependencies
- 20:58Data journey story
- 21:41Recursive lineage helper
- 23:04Visualizing data journeys
- 25:07Downstream impact of changes
- 26:11Detecting orphaned data
- 26:32Key takeaways and closing remarks
related talks
17mOptimizing Data-Driven Decisions
19mDemystifying Modern Data Pipeline Architecture
24mDataEngBytes 2023 - MEL-T3-03 - Akira Wang & Abdi Farah
45mLet's Do Data Lineage in Kafka, Flink and Druid by Tracking Aircraft!
28mBuilding Self-Service Data Platforms
17mDemocratising data analytics
30mData Engineering building your BI infrastructure from scratch
34mColumn-level lineage is coming to the rescue