Scaling with Apache Spark (or a lesson in unintended consequences)
summary
Examines Apache Spark’s scaling tradeoffs with concrete examples: RDD pipelining and laziness speed things up but produce 400‑line stack traces and late failures; reduceByKey over groupByKey often changes a failing job into one that finishes; deterministic partitioning causes key skew that melts nodes and forces manual key-salting; DataFrame pushdown helps but has blind spots and iterative ML on DataFrames performs poorly; ongoing work targets a non‑deterministic partitioner and Apache Arrow integration for cross‑language and GPU performance, improvements not yet fully deployed.
AI-written summary · about
chapters (11)
- 0:00Start
- 8:31Why people come to Spark
- 9:19What's Spark history?
- 15:10Word count (in python)
- 17:23Why laziness is cool (and not)
- 20:06Design part #2: partitioning
- 23:45So what does groupBy Key look like?
- 30:56Design part 3: Arbitrary functions
- 32:35Mini "fix": Datasets (aka DataFrames)
- 34:34And functional style maps
- 38:23Patches Welcome?
related talks
43mBeyond Shuffling: Scaling Apache Spark
41mOptimising Apache Spark and SQL for Improved Performance
50mChallenges that everyone struggles while productionizing Apache Spark workloads
28mAn Apache Spark query's journey through the stack
04mInterview
40mBest Practices and Optimization with the Spark Engine
25mScaling Up Genomics with Spark
01h 18mScalable Data Science with Spark and R