Contracts, Schema Evolution in Data Pipelines
summary
Covers best practices for modeling and managing evolving data schemas in high-volume data processing pipelines, based on Zapr’s experience profiling millions of users for TV consumption. Focuses on how to represent and track fields in pipeline data to avoid uncontrolled field growth and the business and technical benefits of a disciplined schema/contract approach; does not delve into specific technologies, scaling metrics, or performance optimizations.
AI-written summary · about
chapters (11)
- 0:00Start
- 16:28Format of Choice?
- 18:45Advantages of thrift
- 19:40Encoding - Thrift
- 20:09User Contract
- 20:30Forward and Backward Compatibility
- 22:02Full Compatibility (definition)
- 23:28Full Transitive Compatibility via Thrift
- 24:15Long Term Support
- 24:36One Contract for a pipeline - Easy to build
- 25:14Development ease
related talks
20mBuilding Case for Standardized Data Pipeline for Organizational Data
27mLive in the Data Wild West: The Data Contracts Sheriff
31mHelp! My Database Is The Bottleneck
21mApplying Continuous Delivery to Database Schema
30mExpand Contract Pattern for Continuous Delivery of Databases
29mSchemas: The Key To Data Happiness
16mBeyond ETL: Revolutionizing Data Engineering Workflows
19mExpand Contract Pattern for Continuous Delivery of Databases