From UDF to Broadcast Join and Back: The Story of a Spark Optimization

on YouTube ↗48m

" summary

A practical case study of optimizing a Spark job: starting from Python UDFs, the talk examines execution plans and performance pitfalls, then refactors the job into a Broadcast Join and back, showing step-by-step improvements and lessons about Scala vs PySpark and Python UDF performance.

AI-written summary · about

" chapters (10)

1. 0:00Introduction
2. 4:47Problem statement
3. 12:46First Python UDF solution
4. 18:40UDF performance issues
5. 23:24Scala UDF solution
6. 26:38Broadcast Join intermediate
7. 30:31Math model error
8. 32:19Final optimal solution
9. 36:27Benchmark results and conclusions
10. 38:34Q&A

" related talks

• 30mImproving PySpark Performance: Spark performance beyond the JVMHolden Karau · PyCon AU 2017
• 36mNo more struggles with Apache Spark (PySpark) workloads in productionChetan Khatri · PyConZA 2019
• 30mOptimize Spark Jobs Using Its UIRaphaël Luta · Devoxx France 2023
• 34mWorking with Tensorflow from the JVM #bbuzzHolden Karau · Berlin Buzzwords 2018
• 21mAvoiding Spark Pitfalls at ScaleLong Cao · Scale By the Bay 2017
• 29mDataEngBytes 2023 - BRI-T1-03 - Yousry MohamedYousry Mohamed · DataEngBytes Brisbane 2023
• 23mEnriching datasets using Apache Spark solves tracing in financial industryGokul Prabagaren · Indy Cloud Conf 2020
• 36mScaling data workloads using the best of both worlds: pandas and SparkEng & Kwon · PyData Seattle 2023