Introduction to Spark Datasets
summary
Datasets claim to combine relational and functional APIs and deliver measurable wins: benchmarks show Dataset/DataFrame runs far faster than RDDs and a sample representation drops from 33 MB to 7 MB in memory, mostly because the optimizer can operate on schema and serialized data and do partial aggregations. Caveats include GroupByKey is the devil for RDDs, iterative ML loops sometimes explode because the optimizer lacks common‑subexpression recognition, and structured streaming via readStream / structured streaming works but lacks some trigger and watermark features.
AI-written summary · about
chapters (22)
- 0:00Intro
- 3:53What is Spark?
- 5:54The different pieces of Spark
- 6:50Why should we consider Spark SQL?
- 10:37What is the performance like?
- 11:52How is it so fast?
- 12:51How much more space efficient?
- 13:44Getting started
- 16:42Loading some simple JSON data
- 18:28Sample case class for schema
- 18:44Then apply some type magic
- 19:38What do relational transforms look like?
- 20:18Writing a relational transformation
- 23:45What can the optimizer do now?
- 25:44Using Datasets to mix functional & relational style
- 27:40And functional style maps
- 29:49What is DS functional perf like?
- 33:07Build the recipe for each query
- 35:20Windowed operations
- 36:13Window specs
- 36:52Summary: Why to use Datasets
- 38:10The next book.....
related talks
59mSpark 2.0
44mProcessing large data with Apache Spark
01h 18mScalable Data Science with Spark and R
18mLooking under the hood - demystifying data tools
32mWhy Use Scala in a Bigdata Context?
42mScala: The Unpredicted Lingua Franca for Data Science
53mTips and tricks for robust big data applications
36mScaling data workloads using the best of both worlds: pandas and Spark