Dask DataFrame 2.0: Comparison to Spark, DuckDB and Polars
summary
Reimplemented Dask DataFrame claims roughly 20x speedups on representative TPCH queries by combining Arrow strings, a new shuffle algorithm and a query optimizer that does column projection, predicate pushdown, auto‑repartitioning and shuffle‑elision. Benchmarks compare Dask to Spark, DuckDB and Polars across 100 GB–10 TB: Dask becomes competitive with Spark, reads far less data (examples show processing 6 TB while the dataset is 1 TB) and completes many 10 TB queries on a 500 GB cluster where older Dask timed out.
AI-written summary · about
chapters (15)
- 0:00Dask DataFrame 2.0 intro
- 1:34Demo 18 months ago
- 3:27Performance improvements
- 5:12Arrow strings for memory
- 7:19Optimized join algorithms
- 8:44New shuffle algorithm
- 10:30Query planning layer
- 13:21Auto repartitioning
- 16:11Eliminating redundant shuffles
- 18:13Benchmarking setup
- 22:56Dask vs Spark DuckDB Polars
- 26:07Dask vs DuckDB 100GB-10TB
- 28:17Dask vs Spark 1TB-10TB
- 31:06Future improvements
- 33:45Q&A
related talks
30mPandas + Dask DataFrame 2.0 - Comparison to Spark, DuckDB and Polars
37mSpark, Dask, DuckDB, and Polars: Benchmarks
01h 22mAn Introduction to Pandas 2, Polars, and DuckDB
28mAdapting from Spark to Dask: what to expect
34mData Wrangling with DuckDB
31mPolars on Spark
28mArrow revolution in pandas and Dask
35mHow many dataframe libraries do you need to change a lightbulb?