Data Reliability Engineering: Secret Of Our Success
summary
Built a harmonized reporting product for a large publisher using data contracts, CI/runtime validation, and production monitoring; input violations most often are uniqueness and referential integrity issues that arrive at different rates, so a spreadsheet-first contract and an aggregation team align producers and consumers. Validation includes a deep difference check that runs inputs through two pipeline versions to catch unintended changes, and monitoring uses row-count anomalies (std dev alerts) plus Slack alerts; tooling examples: DBT and GCP BigQuery lineage API — the approach reduces incidents, speeds debugging and builds consumer trust.
AI-written summary · about
chapters (10)
- 0:00Introduction to talk
- 1:04Project background and challenges
- 4:16What is a data product
- 6:00Data contracts explained
- 9:30Lessons from implementing contracts
- 13:45Validating data contracts
- 16:37Validating data interpretation
- 19:30Regression testing for data
- 21:37Data monitoring in production
- 28:22Summary and Q&A
related talks
37mData Validation at Scale: Managing Data Quality in Complex Data Pipelines
18mData quality: the key to long term happiness
20mDataEngBytes 2023 PER-T1-10 - Matthew Carter
30mDataEngBytes 2023 - PER-T1-01 - Chad Sanderson
41mDataEngBytes 2023 - BRI-T1-01 - Chad Sanderson
44mData Quality Management
44mPreventing, Diagnosing & Curing Bad Data
45mValidating Big Data Jobs - Stopping Failures before Production