Keeping the “fun” in Apache Spark Datasets and FP
summary
Covers concrete limits and wins of Spark Datasets: RDD→Dataset migration yields roughly 2x storage reduction in the presenter’s tests and faster shuffles via Tungsten but only in specific cases; closure serialization breaks on surprising closures so Spark embeds a problematic closure cleaner; Arrow lets Python UDFs run faster and be called from JVM code (Arrow integration), but structured streaming ships two engines — one high‑throughput engine that can drop data — and ML pipelines still use stringy features and awkward fit semantics. Offers examples (word count, windows, custom aggregates) and asks for fixes contributed to the codebase.
AI-written summary · about
chapters (16)
- 0:00Start
- 0:37Pride Month
- 5:22Why Does Google Cloud Care about Spark
- 7:53What Is Spark
- 9:27Why Do People Come To Spark
- 11:32Resiliency
- 12:04Functional Programming
- 14:56Closure Cleaner Scala
- 15:56Stack Traces
- 16:56Machine Learning Api
- 17:43Data Sets and Data Frames
- 23:57Word Count
- 27:30Custom Aggregates
- 29:08Functions Scala
- 41:05Books on Sparc
- 41:49High-Performance Sparc
related talks
45mScala and the JVM as a Big Data Platform Lessons from Apache Spark
46mIntroduction to Apache Spark with Frameless
01h 12mBig Data in Java World: Apache Spark and Scala. Arch and live coding
47mScala: The Unpredicted Lingua Franca for Data Science
47mScala: The Unpredicted Lingua Franca for Data Science
46mScala and the JVM as a Big Data Platform
40mProductionizing Big Data - Stories from the Trenches
55mUse Spark from Anywhere