Improving PySpark Performance: Spark performance beyond the JVM
summary
PySpark bottlenecks center on the double serialization cost between Python and the JVM and DataFrames move heavy work inside the JVM so Python code runs much faster. Replacing groupByKey with reduceByKey cuts shuffle (example: shuffled read drops from ~48KB to ~11.6KB in a demo) and map-side combine helps. For extreme hotspots the path is to register JVM UDFs or call internal JVM methods; for unreliable clusters prefer checkpoint noisy clusters over naive caching.
AI-written summary · about
chapters (17)
- 0:00PySpark performance beyond JVM
- 3:53Apache Spark overview
- 6:23PySpark architecture pain points
- 8:49Word count serialization cost
- 10:12DataFrames vs RDDs performance
- 12:19Optimizing word count with UDFs
- 14:27Mixing JVM and Python code
- 15:31Internal Spark methods access
- 18:25DataFrame performance best practices
- 20:51Why groupByKey is slow
- 24:29groupByKey vs reduceByKey
- 27:43Sorting without explosions
- 29:13Reductions and treeReduce
- 30:40RDD caching and checkpointing
- 34:11Python libraries in Spark
- 35:36Spark performance resources
- 38:05Q&A
related talks
41mImproving PySpark Performance Spark performance beyond the JVM
50mBig Data Processing and Data Science Environment in Python
43mPySpark in Practice
41mKeynote: Making the Big Data ecosystem work together with Python
24mPySpark - Data processing in Python on top of Apache Spark.
46mInternals of Speeding up PySpark with Arrow
31mBringing the Jewels of the Python World to Scala with Spark
45mIndroduction to the PySpark DataFrame API