Pandas + Dask DataFrame 2.0 - Comparison to Spark, DuckDB and Polars
summary
Dask integrates pandas 2.0, Arrow strings and a rewritten P2P shuffle plus a lightweight logical query planning layer. Benchmarks use TPC-H at 100 GB / 1 TB / 10 TB: Arrow strings cut memory from ~220 GB to ~90 GB and speed string ops up to 10x; partition squashing and planner yields a 19x speedup on a dumb query; on 100 GB DuckDB often wins, on 1–10 TB Dask is competitive and beats Spark on many queries, while DuckDB excels if data fits one node; some queries still hit memory/time failures.
AI-written summary · about
chapters (15)
- 0:00Talk introduction
- 0:46Why Dask DataFrame 2.0 is fast
- 3:15Arrow strings for memory
- 5:00Shuffle algorithms improved
- 7:26Query planning in Dask
- 10:35Partition sizes optimized
- 12:20Reducing merge shuffles
- 14:10Summary of improvements
- 15:35Benchmark setup
- 17:44TPC-H 100 GB results
- 18:48Dask vs DuckDB on 1 TB and 10 TB benchmarks
- 21:58Dask vs Spark benchmarks
- 25:07Benchmark takeaways
- 26:09Future Dask improvements
- 27:17Q&A
related talks
42mDask DataFrame 2.0: Comparison to Spark, DuckDB and Polars
31mThe Arrow revolution in pandas and Dask
37mSpark, Dask, DuckDB, and Polars: Benchmarks
01h 22mAn Introduction to Pandas 2, Polars, and DuckDB
28mArrow revolution in pandas and Dask
34mData Wrangling with DuckDB
30mIs the great dataframe showdown finally over? Enter: polars
35mHow many dataframe libraries do you need to change a lightbulb?