A Brave New World: How New Lineage Technology Can Reduce Data Spend 40%
summary
Claims a multi-layered lineage system that overlays dbt models, Snowflake tables and Tableau dashboards with usage and cost metrics to map which producers feed which consumers and why. Shows concrete tooling: Kafka/Kafka Connect ingestion, Snowflake storage, dbt transforms, Tableau/React consumers. Gives a real case where stopping an off-hours refresh saves ~$20k and argues this approach cuts overall data spend by ~40% by exposing wasted refreshes, idle datasets and unowned expensive pipelines.
AI-written summary · about
chapters (7)
- 0:00Introduction to data lineage
- 1:08Business requirements vs actual usage
- 3:39Personal failure story and cost impact
- 5:26Data product ownership and monitoring
- 8:16Data usage tracking across platforms
- 10:24Lineage-driven cost savings
- 12:32Key takeaways and conclusion
related talks
23mData Team as an Optimisation Problem
01h 17mData Engineering in the AI Era: Building a Robust Foundation for AI Applications
24mDataEngBytes 2023 - MEL-T3-03 - Akira Wang & Abdi Farah
23mDelivering Retail Analytics & Data Projects With Data Created For Purpose
25mRemove Slow, Expensive ETL & Bring Data To The Data Lake For Lower Cost, More Performant Analytics
31mFrom Raw Data to Real Impact: Lessons from Real-World Transformations
39mBuilding a Modern Data Warehouse Without Breaking the Bank
25mPerfecting Our Modern Data Stack Recipe: What We Learned On The Journey At Huel