Designing Resilient Data Pipelines
summary
Shows concrete fixes: collect logs plus pipeline- and task-level metrics to trace a days-long ETL latency jump to a degraded shared resource; encode pipelines as DAGs in declarative files (YAML) so changes live in source control, get code review, static analysis and atomic rollback; make tasks idempotent and enforce immutable inputs so retries, checkpointing and automated recovery work without human cleanup; add validators as quarantine validators (leaf nodes) then promote them into the critical path if stable.
AI-written summary · about
chapters (5)
- 0:00Introduction to pipelines
- 4:58Pipeline observability
- 10:59Reliability and structure
- 14:44Pipelines as code
- 20:44Data validation and Q&A
related talks
45mResilient Predictive Data Pipelines
19mObservability for Data Pipelines: Monitoring, Alerting & Tracing Lineage
43mData Pipelines à la Mode
31mWhen the data levee breaks: Managing cognitive load in data-intensive projects
16mData pipeline automation with built-in metadata and lineage (PRT054)
42mBringing Software Engineering Rigour to Data, DataEngBytes 2022
20mShift-left testing : Building reliable Data Pipelines
21mSoftware Best Practices for Data Pipelines: Form Energy's Multi-Day Storage R&D Pipeline 🌳