The Arrow revolution in pandas and Dask
summary
Pandas and Dask adopt Apache Arrow so string columns become contiguous and memory efficient and parquet reads can be zero‑copy leading to 50–60% memory reduction on string‑heavy data sets. Pandas offers copy and right semantics (available since 2.0) and Arrow string defaults land around 3.0 with an opt‑in in 2.1. Dask replaces task‑based shuffles with P2P shuffling and inserts a query optimizer that prunes reads and projections producing roughly 5–10x on 100 GB and up to 50–100x speedups on larger workloads.
AI-written summary · about
chapters (8)
- 0:00Arrow in pandas and Dask
- 5:02Arrow benefits for data frames
- 10:09Arrow strings in Dask memory
- 13:25Copy-on-write in pandas 3.0
- 19:43Dask DataFrame shuffle improvements
- 24:42Query optimizer for Dask
- 28:32Demos old vs new Dask
- 30:57Enabling new features
related talks
30mPandas + Dask DataFrame 2.0 - Comparison to Spark, DuckDB and Polars
24mPolyglot data with python: Introducing Pandas and Apache Arrow
32mConnecting PyData to other Big Data Landscapes using Arrow and Parquet
46mA 101 in Time Series Analytics with Apache Arrow, Pandas and Parquet
09mBackstage interview
29mBuilding Large Scale ETL Pipelines with Dask
29mFaster PySpark with Apache Arrow
28mHigh speed data from the Lakehouse to DataFrames with Apache Arrow