Scale Data Science by Pandas API on Spark
summary
Shows how Pandas API on Spark runs real pandas code on Apache Spark, explains the internal frame plus Catalyst optimizer rules used, and reports benchmarks: single‑node overhead but a threshold between n=4 and 5 where Spark beats pandas, and sustained linear scaling to 64 nodes in cluster runs. Demonstrates query plans changing (left join to inner, adaptive switch to broadcast) and notes future work including Spark Connect removing the JVM client and ~85% API coverage with ongoing performance improvements.
AI-written summary · about
chapters (8)
- 0:00pandas API on Spark intro
- 4:50pandas vs Spark DataFrames
- 6:18Internals of pandas on Spark
- 13:23Scaling data science
- 16:42Performance comparison
- 19:59Benchmarking clusters
- 21:00Catalyst query optimization
- 23:09Future of pandas API
related talks
34mPandas vs Koalas: The Ultimate Showdown!
24mPySpark - Data processing in Python on top of Apache Spark.
50mBig Data Processing and Data Science Environment in Python
01h 13mFrom Pandas to PySpark
45mIndroduction to the PySpark DataFrame API
38mHow to incrementally scale existing workflows on Spark, Dask or Ray?
26mBodo: Supercomputing-Like Performance and Scale for Python/Pandas
31mComparing the Different Ways to Scale Python and Pandas Code