Optimising Apache Spark and SQL for Improved Performance
summary
Shows concrete failures and fixes: Spark 3 still leaves cases where a single partition swallows most data — 100 GB input becomes a 67 GB partition across 1,000 buckets — causing OOM and retries; solutions include day-based repartitioning (adds edge-record stitching) and salting to split joins for cartesian explosion so billions of pairs run across many tasks; warns lazy random-number generation causes train/validation leakage and advises materializing results; claims and examples are benchmarked and reproducible.
AI-written summary · about
chapters (17)
- 0:00Spark performance intro
- 1:11Spark versions history
- 2:34Spark SQL limitations
- 3:41Skewed data with window functions
- 13:02Splitting skewed partitions
- 16:10Avoiding cartesian joins
- 22:20Lazy eval pitfalls
- 26:36Optimization vs business needs
- 28:01Q&A on salting and clusters
- 33:02Learning Spark optimization
- 34:30Adaptive Query Execution limitations
- 35:37Aggregation vs self-join
- 36:22Spark runtime relevance
- 37:03Salting in Python workflows
- 38:29Alternatives to random splitting
- 39:11RPA terminology
- 39:31Salting for data skew
related talks
50mChallenges that everyone struggles while productionizing Apache Spark workloads
43mBeyond Shuffling: Scaling Apache Spark
40mScaling with Apache Spark (or a lesson in unintended consequences)
15mOptimizing Query Performance in Big Data Systems
40mBest Practices and Optimization with the Spark Engine
28mAn Apache Spark query's journey through the stack
39mStorage-Partitioned Join for Apache Spark
04mKuassi Mensah Interview