Breaking down large tests with a Saff Squeeze
summary
A PySpark pipeline contains a one-line bug (forgotten parameter causing an inner join instead of a left anti-join) that corrupts two months of data, costs a team of eight five days to locate, three days to fix and seven days to backfill. A large integration test with 100% coverage hides the fault because it overwrites expected JSON. Using the Saf Squeeze method duplicates the test, makes a pinpoint assertion, inlines transforms and reduces inputs to tiny DataFrames, producing small, fast tests that catch both behaviors and save days.
AI-written summary · about
chapters (20)
- 0:00Introduction
- 0:42Talk overview
- 1:23Story of a Xenopus frog
- 2:46Manual ETL pipeline errors
- 3:50Automated tests power
- 4:11Pandemic recovery bug
- 7:00Root cause investigation
- 9:50Failing automated tests
- 11:15Jay Bazuzi wisdom
- 12:38Why large tests slow dev
- 15:51Saf Squeeze technique
- 17:14Duplicating test
- 19:22Breaking test into parts
- 21:50Squeezing bottom
- 25:01Squeezing top
- 27:30Effort vs improvement
- 28:36Testing second behavior
- 30:21Benefits at Yelp
- 31:44Closing remarks
- 32:05Q&A
related talks
39mWhat Can Go Wrong With Testing an ETL Pipeline?
25mTesting Data Pipelines
20mShift-left testing : Building reliable Data Pipelines
46mSimplifying Testing of Spark Applications
45mDo those numbers look right? How to ensure quality, and avoid inaccuracies in your data insights.
39mTesting Big Data Applications (Spark, Dask, and Ray)
28mHow to not pull your hair out while providing data to the business
34mTesting Big Data Applications (Spark, Dask, and Ray)