Spark Datasets: why they aren't great and what could be done
summary
Argues Spark Datasets trade type safety for optimization visibility and shows a working alternative: a Quill-based quoted DSL that converts Scala code into a SQL-like AST so the Spark SQL engine can optimize while keeping static types. Demonstrates a Spark integration that generates queries at compile time, supports composable monadic joins and concatMap instead of opaque flatMap, and stays lazy. Prototype works but is young: compilation fails for non-translatable code and feature coverage versus native SQL is incomplete.
AI-written summary · about
chapters (4)
related talks
47mKeeping the fun in Apache Spark Datasets and FP
06mInterview
01h 36mSpark for Scala Developers
16mFrameless: A More Well-Typed Interface for Spark
24mImproving interactive querying experience on Spark SQL
31mKotlin for Apache Spark: Why the world needs one more API for Spark
55mSpark Architecture
05mInterview