Internals of Speeding up PySpark with Arrow
summary
Explains exactly what changes: PySpark uses Apache Arrow record batches to avoid pickling, enabling record batches / zero-copy conversion between JVM and Python and switching Python UDFs from slow pickled rows to fast columnar execution. Presents numbers from benchmarks — typical 10–70× speedups — shows the config flip that enables it, describes the Python worker model (one worker per executor process), and demos groupby/pandas-UDF examples plus links to slides and code.
AI-written summary · about
chapters (10)
- 0:00Start
- 5:09HOW DOES PANDAS MANAGE COLUMNAR DATA?
- 7:55HOW DOES ARROW MANAGE COLUMNAR DATA?
- 11:30HOW DOES SPARK WORK?
- 12:22AND A DISTRIBUTED STORAGE
- 12:56THE DRIVER REQUESTS RESOURCES FROM THE CLUSTER MANAGER TO RUN TASKS
- 19:43PYSPARK OFFERS A PYTHON API TO THE SCALA CORE OF SPARK
- 28:12YES, FOR SPARK DATAFRAME
- 31:30QUICK EXAMPLES
- 38:48QUESTIONS?
related talks
29mFaster PySpark with Apache Arrow
46mA 101 in Time Series Analytics with Apache Arrow, Pandas and Parquet
24mPolyglot data with python: Introducing Pandas and Apache Arrow
32mConnecting PyData to other Big Data Landscapes using Arrow and Parquet
41mImproving PySpark Performance Spark performance beyond the JVM
31mThe Arrow revolution in pandas and Dask
41mKeynote: Making the Big Data ecosystem work together with Python
41mFaster SQL with pandas and Apache Arrow