Who Really Uses Dataform? Spoiler: We Do, and Here’s Why!
summary
Pet Circle migrates SQL transforms onto Dataform on BigQuery, tied to GitHub and GitHub Runners, with Cloud Scheduler, Firestore, Cloud Functions and Google Workflows for orchestration and Terraform for infra. A custom config generator produces model configs and tags; table dependencies, assertions and data quality checks run automatically. Deployment time drops from four hours to five deployments a day with clearer downstream impact and centralized quality alerts, showing the platform works when paired with upfront modeling, source control and lightweight serverless glue.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 2:31Why Pet Circle needed Dataform
- 5:00Dataform vs DBT
- 9:59Dataform at Pet Circle
- 11:21Key features that worked
- 13:24Managing dependencies
- 15:27Data quality checks
- 16:30Automating config generation
- 17:52Data quality reporting
- 19:16One year later: Results
- 20:42Should you use Dataform?
- 21:03Final takeaways and conclusion
related talks
51mNatalie Godec DevopsDays Geneva 2020
29mManaging your Data Assets: The interview
48mBuilding a data mesh
31mData Mesh & Data Vault
38mLesson learned implementing a large-scale data mesh transformation
35mBuilding a data analytics platform on strong opinions held loosely
43mFostering a Data-Driven Culture
23mData Team as an Optimisation Problem