Asset based orchestration: the future of data orchestration
summary
Claims that shifting from task/DAG scheduling to asset-based orchestration fixes dbt scale problems by importing assets across repos, building a global dependency graph, and triggering materialization from leaf freshness targets. Demo uses Dexter to ingest assets from dbt and Airbyte, sets a global 9-hour freshness SLA with a single model overwritten to 5-minute freshness, and runs a sensor every 30 seconds. Approach is demonstrated in a repo but remains exploratory at Canva, not yet production-deployed.
AI-written summary · about
chapters (5)
- 0:00Introduction
- 0:44Defining the orchestration problem
- 8:07Introducing asset aware orchestration
- 13:23Demo of Dexter tool
- 21:31Conclusion and future outlook
related talks
22mDeveloping, Deploying And Scaling A Global Asset Management Capability
49mdbt: A Software Engineering Approach To The Analytics Workflow
05mLightning Talk: Ditching Data Pipelines: why treating Data as Assets is the best...
26mModern Data Orchestrators
23mDelivering Modern Data Stacks With dbt to Operationalise Analytics
19mRobust CI Workflow with dbt for Data Warehouse Optimization
25mGone Streaming: dbt+Materialize
10mLightning Talk: We Built a Unified AI Asset Catalog in Backstage