Failing Fast with Data: Bringing CI to the Land of Data
summary
Claims that CI for data pipelines works are backed by specifics: a YAML config plus a JSON schema registry generates auto-generated schema tests that run the actual PySpark code with synthetic rows, catching schema breaks and missing columns. That POC becomes org-wide for forecasting (2–10M predictions/day), shrinks a six‑month data‑warehouse change path to about 1 hour change lead time, and leaves CI bottlenecks around a 20 minutes CI runtime while enabling feature toggles and shadow experiments in production.
AI-written summary · about
chapters (28)
- 0:00Introduction
- 1:05What this talk is about
- 1:42Why am I here
- 2:39Characters
- 3:16Frustration
- 3:55The reality
- 4:56The red thing
- 5:26What we want to do
- 5:48Storytime
- 6:33Data Quality
- 7:08Teaser
- 7:23Critical Path
- 8:43History of Data
- 9:20What could go wrong
- 10:53Will my job run
- 13:38Demo
- 16:23Data as code
- 17:32Pitfalls
- 18:26Productivity handcuffs
- 19:19Do more work
- 20:02Learning in production
- 20:44Data dependencies
- 21:08Splitting data sets
- 21:29Updating schema
- 22:20Make sure they run
- 22:42Its an ideal world
- 23:47How do we do this
- 26:05Summary
related talks
38mKeeping pace with data infrastructure so it doesn't outrun you
16mBeyond ETL: Revolutionizing Data Engineering Workflows
20mBuilding a Data Pipeline with Testing in Mind
30mDevOps For Data Teams
36mData pipelines engineering made simple with Scala
26mDataOps: Accelerating Digital Transformation
28mBuilding Self-Service Data Platforms
31mHow to quickly build Data Pipelines for Data Scientists