End-to-end pipeline agility
summary
Quantitative claim: adapting downstream pipelines ranges from median 6 months to some teams doing it in less than a day, a gap of 100x–1000x. The setup uses Luigi and a test harness called the Warp Zone that rewrites DAG targets to local disks for CI, enabling end-to-end pipeline tests. Schema metaprogramming with Scala Meta generates Avro, privacy variants and transforms so schema-on-write stays statistically typed, cutting boilerplate. Swapping Spark for plain Scala collections speeds iterative tests; continuous deploy and dataset versioning (frozen-lake) make changes safe.
AI-written summary · about
chapters (9)
- 0:00Introduction and context
- 5:20Data agility gap and impact
- 10:45Workflow orchestration fundamentals
- 17:36Testing pipelines and workflows
- 23:39Data versioning and immutability
- 26:27Schema management strategies
- 35:44Code generation for agility
- 36:46Speed of feedback loops
- 37:29Q&A session
related talks
31mOne Data Pipeline to Rule Them All
24mDistinguishing features of production quality pipelines
20mShift-left testing : Building reliable Data Pipelines
41mSpeeding Up Data Pipeline Construction in Apache Spark
38mEnabling Big Data and Machine Learning for the Masses...
28mDynamic Data Pipelining with Luigi
39mScaling ETL Migrations to 300x with AI
43mData Pipelines à la Mode