How to maximally parallelize the entire pandas API
summary
UC Berkeley research yields Modin, a distributed drop-in replacement for pandas with a VLDB paper describing a novel parallelization scheme, lazy metadata materialization and a 16-operator data‑frame algebra. The implementation composes API → query compiler → algebra → execution layers and runs on Ray/Dask/Spark backends, automatically partitions frames and defaults to pandas when needed. Benchmarks report roughly 95% pandas API coverage and consistent speedups on ops like fillna and median, so it actually scales.
AI-written summary · about
chapters (17)
- 0:00Introduction
- 2:13Pandas scaling limits
- 3:16Data frame decomposition
- 5:23Metadata independence
- 6:25Data frame algebra operators
- 7:06Modin architecture layers
- 8:08Modin vs competitors
- 10:12Modin on Databricks
- 12:19Modin on Snowflake
- 13:02ML integration with Modin
- 14:44Modin with Ray and Dask
- 16:09Unsupported pandas APIs
- 17:55Pandas code changes
- 19:42Query optimization
- 20:32Modin vs Dask DataFrames
- 22:15Query plan visualization
- 23:20Q&A
related talks
38mScaling Interactive Pandas Workflows with Modin
29mLearning to Scale Data Science, Machine Learning, and Pandas with Ray and Modin
26mBodo: Supercomputing-Like Performance and Scale for Python/Pandas
01h 17mPandas from the Inside / "Big Pandas"
05mPandas on DuckDB with Ponder
14mKeynote: Scalable Python We Can Afford
39mAccelerated Data Science Libraries
37mpandas at a Crossroads, the Past, Present, and Future