Robust CI Workflow with dbt for Data Warehouse Optimization
summary
Midsize company (100 employees, ~50 engineers) with a two-person data team builds a dbt-centered CI that enforces git-only changes, dynamic per-model Airflow DAG tasks for partial reruns, SQLFluff linting and autofmt, and dbt-expectations style unit tests plus metadata checks so every model has an owner and column descriptions. Dev branches get prefixed, isolated datasets with 72-hour dev datasets lifecycle; tests use partition pruning (last two weeks) or dbt’s flag to read production artifacts to cut scan volume. Contracts run pre-materialization and model versioning with view-based releases prevent accidental production breaks.
AI-written summary · about
chapters (13)
- 0:00Robust CI with dbt intro
- 1:54Why we chose dbt
- 3:19Dynamic DAG for partial reruns
- 4:22Git-driven CI for SQL
- 6:09SQL style with SQLFluff
- 7:31Multi-level testing in dbt
- 9:16Metadata and ownership
- 10:23Dev vs production separation
- 12:11Dev environment lifecycle
- 12:54Reducing data scan
- 14:38Breaking changes with contracts
- 16:00Model versioning
- 17:24Key learnings
related talks
49mdbt: A Software Engineering Approach To The Analytics Workflow
39mBuilding a Modern Data Warehouse Without Breaking the Bank
23mDelivering Modern Data Stacks With dbt to Operationalise Analytics
21mSnowflake and dbt -- Our Journey to the Cloud
28mHow Open Source Enables Innovation A Case Study with dbt and Lightdash
20mDataEngBytes 2023 PER-T1-10 - Matthew Carter
31mHow to Implement DataOps using
01h 07mHow to Count Things with dbt - Data Build Tool