Keeping the fun in Apache Spark Datasets and FP
summary
Apache Spark Datasets add typed functional operators on top of faster, SQL-like DataFrames but lose type safety at groupBy and custom aggregation points, causing runtime Any/mutable Row pain; RDDs still win for some iterative ML and pathological cases. The DataFrame V1 choice trades untyped SQL ergonomics for speed (slides claim ~50% storage/5–10% runtime gains), and the Arrow/pandas UDF story promises big speedups (the 242× claim is exaggerated) but requires boilerplate to call Python from Scala. Frameless and user-defined aggregators can restore types, but current ML Pipelines remain runtime-schema validated and brittle for long jobs.
AI-written summary · about
chapters (12)
- 0:00Start
- 4:22What Spark Is
- 4:58Machine Learning
- 6:48What Is the Magic of Spark
- 14:55The New Api'S
- 15:17Data Frames
- 18:57Triple Equals Operator
- 23:33Windowed Operations
- 31:38Apache Aero
- 32:43Python
- 37:00Main Takeaways
- 40:05Closing Panel
related talks
06mInterview
23mSpark Datasets: why they aren't great and what could be done
30mImproving PySpark Performance: Spark performance beyond the JVM
16mFrameless: A More Well-Typed Interface for Spark
30mIs the great dataframe showdown finally over? Enter: polars
55mSpark Architecture
03mFred Reiss Interview
21mFP for Data Science For Loops Considered Harmful