Data pipeline automation with built-in metadata and lineage (PRT054)
summary
Conference claim: 95% of teams at or over capacity and an independent survey of 500 data experts finds only 3.5% currently invest in automation while 85% intend to within 12 months. The talk explains flipping imperative ETL scheduling into a declarative control plane (logic/control/data planes), binding code and data with SHA-based fingerprinting to track trillions of records, millions of partitions and thousands of transforms, enabling automated restarts, smart backfills, efficient incremental propagation and mid‑pipeline SHA skips that avoid reprocessing unchanged data.
AI-written summary · about
chapters (10)
- 0:00Introduction and overview
- 0:43The new scale challenge
- 1:25Need for automated systems
- 3:32Shifting to declarative
- 5:44Control plane architecture
- 7:05Metadata driven automation
- 8:44Data tracking and layers
- 10:27Code and data hashing
- 12:11System integrity checks
- 13:26Real-world scenarios
related talks
30mDesigning Resilient Data Pipelines
38mKeeping pace with data infrastructure so it doesn't outrun you
27mData team strategies: unleashing your team
27mThe Sanity Check: Observability for AI Data Pipelines
11mTransforming Problem-Solving with Data-Driven Insights
42mBringing Software Engineering Rigour to Data, DataEngBytes 2022
14mData Lifecycle Management as Foundational Pillar to AI Agents
41mModern Data Architectures – What They Are and How to Implement?