Scaling data workloads using the best of both worlds: pandas and Spark
summary
Pandas API on Spark lets code scale by swapping import pandas for pyspark.pandas and shows near linear speedup as nodes increase in their benchmarks, with API coverage around 70–80% today. Pandas Function APIs (Spark 3.0+) run groupby-apply workloads — e.g., training per-device ML models in parallel — but require salting to avoid OOM on huge groups. Spark Connect decouples driver and client, ships plans in Arrow, and is progressing toward full pandas/UDF support in upcoming Spark 3.4+.
AI-written summary · about
chapters (6)
- 0:00Introduction
- 1:05Scaling pandas with minimal changes
- 6:39Pandas API on Spark
- 12:59Custom pandas functions in Spark
- 22:57Spark Connect for development
- 33:32Q&A on pandas UDFs
related talks
31mComparing the Different Ways to Scale Python and Pandas Code
42mScalable Data Science in Python and R on Apache Spark
26mLarge Scale Data Validation (with Spark and Dask)
53mEnterprise-Ready Python for High-Performance Data Teams
32mDataFrames: scaling up and out
29mScaling Python-Powered Machine Learning With Snowflake
30mSpeed Up Your Spark Data Processing with a GPU
44mData Science at Scale with Apache Spark