KONTEXTFREI: A NEW APPROACH TO TESTABLE SPARK APPLICATIONS
summary
Claims a concrete fix: kontextfrei is a small Scala library that abstracts over Spark RDDs via a type class called DecCollectionOps so business logic compiles and runs either on fast in‑memory Scala collections (for ScalaCheck property tests) or on RDDs with a local SparkContext for integration. It implements map/filter and custom map‑side broadcast joins, lacks general accumulators/broadcast support and dataset/streaming adapters, and reports production use restoring the quick TDD feedback loop.
AI-written summary · about
chapters (9)
- 0:00Start
- 0:50Recap
- 2:40Testing Spark Apps
- 3:47Property Based Testing
- 8:59Design Goal
- 13:19Generic Collecting Instances
- 18:09Alternative Designs
- 21:12Broadcasting Joint Operation
- 21:57Advanced Scada Features
related talks
18mSimple and Effective Spark Tests with Spark-Test
46mSimplifying Testing of Spark Applications
39mTesting Big Data Applications (Spark, Dask, and Ray)
34mTesting Big Data Applications (Spark, Dask, and Ray)
39m"Unit tests for Data" with Deequ
50mSimplifying cloud-native app testing across environments
01h 11mKafka streams testing: A deep dive
15mAn Empty Kafka is a Sad Kafka