Spark, Dask, DuckDB, and Polars: Benchmarks
summary
Benchmarks run Spark, Dask, DuckDB and Polars on scales from 10 GiB to 10 TiB using TPC-H on a local M1 MacBook Pro and AWS instances, with orchestration via Coiled. Key findings: no tool wins consistently; DuckDB performs admirably across many scales and sustains ~1–2 GB/s S3 reads on a big node, Dask completes all queries across scales by aggregating bandwidth (peaking near 7 GB/s), Polars is lightning fast on single machines but struggles at large scale, and Spark often needs careful tuning and fails to finish 10 TiB runs under an hour. S3 bandwidth is the dominant bottleneck (per-node ~50–200 MB/s).
AI-written summary · about
chapters (22)
- 0:00Introduction
- 1:31Why compare these systems
- 5:28TPC-H overview
- 7:12Avoiding benchmark bias
- 10:24Local 10GB results
- 11:49Cloud 10GB results
- 12:53Scaling to 100GB and 1TB
- 13:35Analyzing results
- 16:23Cloud bottlenecks
- 18:09Inter-machine communication costs
- 21:02DuckDB performance
- 21:45Coiled scaling
- 25:22Spark on Coiled
- 26:47Dask roadmap
- 27:33Q&A
- 27:55S3 networking
- 29:40Dask goals
- 31:06New data types
- 32:29Data warehouse comparison
- 33:55Hardware relevance
- 34:58Arrow S3 improvements
- 36:01Closing thoughts
related talks
42mDask DataFrame 2.0: Comparison to Spark, DuckDB and Polars
30mPandas + Dask DataFrame 2.0 - Comparison to Spark, DuckDB and Polars
30mSparkless Patterns: The New Data Stack and Modern Data Architecture
19mSpark vs. Accelerated Compute: A Balanced Look at Data Processing Strategies
35mHow many dataframe libraries do you need to change a lightbulb?
28mAnalyze data easily with duckdb - and the implications on data architectures
28mAdapting from Spark to Dask: what to expect
31mPolars on Spark