Modern Data Pipelines Testing Techniques: A Visual Guide
summary
Claim: disciplined testing turns fragile scripts into reliable pipelines by shortening feedback loops to under five minutes when possible. Shows an end-to-end characterization test that runs a 10% stable input sample and compares summary stats to stable output, describes an audit-publish pattern with prod audit and rejected tables, and lists five pillars for data quality tooling (profilers, expectation stores, analyzers, metric stores, reports). Demonstrates using snapshot/clone/time-travel or lightweight clones to run medium tests locally and prevent destructive production writes.
AI-written summary · about
chapters (15)
- 0:00Modern data pipeline testing intro
- 1:59CI/CD for data reporting
- 4:10Common data pipeline failures
- 6:43TDD and quality of life
- 9:53Continuous delivery in data engineering
- 12:04Unit vs integration testing
- 15:14Audit publish pattern
- 17:24Data quality tooling pillars
- 19:32Testing legacy pipelines
- 24:44Test data management
- 28:42Snapshots and time travel
- 32:19Local testing and self-service
- 35:12Designing for testability
- 36:37Data Dev scorecard
- 38:01Conclusion and resources
related talks
25mTesting Data Pipelines
20mShift-left testing : Building reliable Data Pipelines
20mBuilding a Data Pipeline with Testing in Mind
14mDesigning Secure, Compliant, & Scalable Test Data Pipelines for Enterprise
23mChallenges of database testing
27mExploring a DevOps Transformation Like a Tester
26mThe Impact of Testing on a DevOps Pipeline
30mHow to Test BigData/ETL Solutions?