Validating Big Data Jobs - Stopping Failures before Production
summary
Big data pipelines use Apache Spark and Apache BEAM plus helper libs (spark-validator, counters, property-based generators) to run automated schema and semantic checks against historical runs and to gate deployments; about ~40% of teams auto-deploy and ~15.3% report serious outages from jobs. Practical techniques include accumulators/counters (noting Spark’s non-deterministic counting), delta comparisons to prior outputs (expensive joins), automated model canarying with A/B metrics and rollback, and explicit rules that abort deployment when thresholds (e.g., >10% parse errors) trigger.
AI-written summary · about
chapters (36)
- 0:00Intro
- 0:38Agenda
- 2:33Why should we validate
- 3:55Statistics
- 5:03Distributed Systems
- 5:29Notebooks
- 5:47Why dont we validate
- 6:58What happens when we dont validate
- 9:55Simple unit tests
- 10:32Not good enough
- 11:04Data generation
- 12:04Dr Evil
- 12:20Its a really good show
- 12:37How do we validate
- 13:30Making sure your assumptions hold
- 17:04Input Schema Validation
- 20:16Exceptions
- 20:55Spark validator library
- 22:42Data change
- 26:02Upgrades
- 28:29Machine Learning
- 29:30Deploying Models
- 30:01In Practice
- 31:28Statistical Tests
- 31:46TLDR
- 32:22Convergence
- 32:48Crossvalidation
- 34:08Dont be evil
- 34:58Metrics
- 35:49Update your model
- 36:21Ending notes
- 38:13Related tools
- 38:37Copying XML
- 38:50Copying PR
- 39:28Questions
- 39:53Sponsors
related talks
40mValidating Big Data Pipelines & ML (w Spark & Beam)
37mData Validation at Scale: Managing Data Quality in Complex Data Pipelines
42mDeciphering the way data is tested
24mDistinguishing features of production quality pipelines
40mProductionizing Big Data - Stories from the Trenches
03mDBTB INT Prakash Chockalingam r
30mHow to Test BigData/ETL Solutions?
40mPerformance Testing and Big Data