DataEngBytes 2023 - BRI-T1-01 - Chad Sanderson

on YouTube ↗41m

" summary

Argues that API-style data contracts — a YAML contract spec plus CI/CD PR checks tied to schema, semantics and SLAs — prevent churn and costly outages at Convoy, catching ~50% of quality incidents before production and avoiding six‑figure failures caused by dropped columns. Implementation uses a Git repo of contracts, tiered enforcement (0/1/2), Amundsen catalog enrichment, and subscriber alerts; start vertically on business‑critical pipelines and expand incrementally.

AI-written summary · about

" chapters (15)

1. 0:00Data quality for AI
2. 2:34Data quality critical for AI
3. 5:46Root causes of data quality issues
4. 8:57Human challenges behind poor data quality
5. 12:54Why current solutions fail
6. 16:06What is a data contract
7. 21:31Benefits of data contracts
8. 22:13Implementing data contracts
9. 26:03Starting data contract rollouts
10. 29:36Changing culture with data contracts
11. 33:10Data contracts with SaaS
12. 35:39Challenges for large orgs
13. 37:27Data contracts vs data mesh
14. 38:33Balancing quality and velocity
15. 39:17Scaling data contracts

" related talks

• 30mDataEngBytes 2023 - PER-T1-01 - Chad SandersonChad Sanderson · DataEngBytes Perth 2023
• 28mDataEngBytes 2023 - SYD-T1-03 - Chad SandersonChad Sanderson · DataEngBytes Sydney 2023
• 20mDataEngBytes 2023 PER-T1-10 - Matthew CarterMatthew Carter · DataEngBytes Perth 2023
• 21mDataEngBytes 2023 - SYD-T3-04 - Abhinav GoyalAbhinav Goyal · DataEngBytes Sydney 2023
• 27mLive in the Data Wild West: The Data Contracts SheriffTal Peretz · hayaData 2024
• 31mData Reliability Engineering: Secret Of Our SuccessAkshay Karle, Carmen Mardiros · SREday 2023 London
• 20mData Contracts: Keep Your Weekend Work-Free!Nick Pelikan · posit::conf(2024)
• 26mData contracts: driving data culture changeAndrew Jones · Data Teams Summit 2023