Polars on Spark
summary
Claims that combining PySpark Arrow UDFs with Polars yields real speed and memory wins: Polars is a Rust DataFrame that reads Arrow without copying, enabling zero-copy Arrow interoperability. Benchmarks use TPC-DS SF100 (≈288M rows) on a four‑worker cluster (61 GB RAM each) and show pandas UDFs at ~193s versus Polars on Arrow UDFs at 75.5s — about a 2.5× speedup. Per‑batch memory drops from ~425 MB to ~275 MB, and the approach produces working, reproducible pipelines with Polars’ ergonomic API.
AI-written summary · about
chapters (12)
- 0:00Polars on Spark intro
- 1:47What is Apache Spark and PySpark
- 5:21Python UDFs challenges in Spark
- 7:25Evolution of PySpark UDFs
- 8:28Pandas UDFs and Arrow
- 12:49Arrow UDFs limitations
- 15:17Polars for Spark UDFs
- 17:53Polars UDFs performance benchmarks
- 20:04Demo: image processing
- 25:01Future PySpark UDFs improvements
- 26:49Key takeaways and best practices
- 28:21Q&A
related talks
44mPolars; DataFrames in the multi-core era
01h 22mAn Introduction to Pandas 2, Polars, and DuckDB
29mFaster PySpark with Apache Arrow
34mBridging Pandas and Polars
42mDask DataFrame 2.0: Comparison to Spark, DuckDB and Polars
32mThe PyArrow revolution in Pandas
30mPandas + Dask DataFrame 2.0 - Comparison to Spark, DuckDB and Polars
25mSpeed Up Data Access with PyArrow Apache Arrow Data is the new API