Live in the Data Wild West: The Data Contracts Sheriff
summary
Claims a concrete platform that enforces data contracts across ingestion, streaming and analytics, handling schema evolution and preventing bad writes into Delta tables on S3. Uses a schema registry + GitHub Actions to block type-breaks, Kafka + Avro for transport, dbt models and DBT config-driven schemas, a company-wide catalog via DataHub, and automated checks with Elementary and Great Expectations. Faces 463 exabytes/day scale and cites $3.1T estimated US damage from bad data; the pipeline surfaces failing tests in the catalog, opens PR reviews for consumers, and blocks unsafe merges.
AI-written summary · about
chapters (10)
- 0:00Welcome to Data Wild West
- 2:35Introducing Data Contracts
- 4:02Why Data Contracts Matter
- 6:12Data Flow and Schema Registry
- 8:43Handling Schema Changes
- 12:20Data Validation Mechanisms
- 16:23Business Table Creation
- 20:22One Public API Solution
- 22:59Ensuring Data Quality
- 24:49Summary and Key Takeaways
related talks
28mDataEngBytes 2023 - SYD-T1-03 - Chad Sanderson
50mData Contracts VS Data Chaos: A Survival Story
20mData Contracts: Keep Your Weekend Work-Free!
41mDataEngBytes 2023 - BRI-T1-01 - Chad Sanderson
49mTransforming Analytical Data with Event-Driven Architecture
26mData contracts: driving data culture change
29mManaging your Data Assets: The interview
31mOne Data Pipeline to Rule Them All