Harnessing E2E automation to gatekeep data pipeline
summary
A reusable E2E framework uses the existing orchestrator as the test runner to run a day-one/day-two ingestion pattern, auto-clean/setup, trigger pipelines, then compare ingested parquet/hive outputs and schemas against backed-up expected results; built in about 30 days and shrinks a manual 5‑day regression that covered 50 scenarios/1,500 assertions to about two hours, runs daily (200+ cycles), surfaces 4 regressions early, and enables 10+ prod deploys with zero manual regression by being debug friendly and data driven.
AI-written summary · about
chapters (22)
- 0:00Intro
- 2:58Data Pipeline Architecture View
- 5:16Data Source type: Batch vs Stream
- 5:48Factors Leading to Multiple Pipelines
- 6:26Spiralling of Test Scenarios
- 6:51Challenges in manual regression
- 7:42Effort to do Manual Regression...
- 8:12Manual regression becomes...
- 8:37Inefficacy of existing Test Layers
- 9:50In Data pipeline, BUGs can cost..
- 13:44E Test Approach
- 14:08E2E Test: Setup Day 2 Data
- 15:34E2E Test: Full cycle...
- 16:26Test Runner
- 17:38Key Features...
- 17:56#1 Debug Friendly
- 18:17#2 Lean, Fast & Mean...
- 19:03Data driven & Extendable
- 19:27Deeply Assertive
- 19:48Maximise Reuse
- 20:22With Automated Regression...
- 21:02Highlights
related talks
39mWhat Can Go Wrong With Testing an ETL Pipeline?
30mHow to Test BigData/ETL Solutions?
14mDesigning Secure, Compliant, & Scalable Test Data Pipelines for Enterprise
20mDataEngBytes 2023 PER-T1-10 - Matthew Carter
20mBuilding a Data Pipeline with Testing in Mind
16mData pipeline automation with built-in metadata and lineage (PRT054)
25mTesting Data Pipelines
42mEnd-to-end pipeline agility