Arrow revolution in pandas and Dask
summary
Claims are concrete: Arrow-backed strings cut memory (example: three string columns drop ~220 GB to ~100 GB), copy and right turns defensive copies into a fast default (a 3M-row reshape drops from 2.5 s to 13.7 ms ≈ 200×), and Dask’s P2P shuffle lets shuffles run at constant memory so a 10 TB job runs on modest RAM with 30 workers. A logical query optimizer removes huge unnecessary work (62,000 tasks → ~427) and the demos show the new pipeline actually finishes far faster.
AI-written summary · about
chapters (5)
- 0:00Arrow in pandas and Dask
- 4:15Arrow-backed strings and missing values
- 9:28Copy-on-write for DataFrame updates
- 14:05Dask DataFrame with Arrow
- 22:42Enable new Arrow features
related talks
30mPandas + Dask DataFrame 2.0 - Comparison to Spark, DuckDB and Polars
32mThe PyArrow revolution in Pandas
34mLeveraging Apache Arrow for ML Workflows
25mSpeed Up Data Access with PyArrow Apache Arrow Data is the new API
36mEmbrace the chaos: Composable data systems with fewer asterisks
33mArrow in Flight - New Developments in Data Connectivity
32mVectorized Query Processing for CPUs using Apache Arrow
42mDask DataFrame 2.0: Comparison to Spark, DuckDB and Polars