Adapting from Spark to Dask: what to expect
summary
Benchmarks show concrete wins: on a laptop 100M rows runs ~3 minutes on Spark versus ~40 seconds with Dask, and 1B rows drops from ~45 minutes to ~16 minutes; on a small EMR cluster the 1B job goes from ~12 minutes (Spark) to ~9–11 minutes (Dask) depending on worker placement. Custom aggregation requires a Scala UDAF for Spark but a Python chunk/aggregate pair for Dask; output JSON writing needs a workaround in Dask; deployment shifts from YARN/bootstrap scripts to Kubernetes and Docker. Dask gives pandas-like API and no Java stack traces, with caveats around complex nested Parquet, limited built-in aggregations, and evolving tooling.
AI-written summary · about
chapters (24)
- 0:00Intro
- 1:15How does this make you feel?
- 2:32What is Dask?
- 4:22Input Parquet record
- 5:03Aggregated JSON record
- 6:06Input data structure
- 8:20Aggregate with Spark (SQL)
- 8:47Spark needs a custom aggregation
- 9:30Aggregate with Dask
- 10:21Dask needs two custom aggregations
- 11:18At this point...
- 12:02Transformation
- 12:19Transform with Spark
- 13:05Transform with Dask
- 13:15Transform one record with Dask
- 13:46Write JSON with Dask
- 15:23Benchmarking setup
- 16:06Performance (YARN)
- 17:27Cluster deployment with Spark (AWS)
- 18:22Cluster deployment with Dask (Kubernetes)
- 19:52Speaking of promises
- 20:59The bad
- 23:08The full code
- 23:25Questions?
related talks
25mDistributed data workflows: PySpark vs Dask
01h 03mDask: Next Steps in Parallel Python
42mDask DataFrame 2.0: Comparison to Spark, DuckDB and Polars
01h 26mDask tutorial
01h 22mUsing Dask for Parallel Computing in Python
30mPandas + Dask DataFrame 2.0 - Comparison to Spark, DuckDB and Polars
01h 23mData Science at Scale with Dask
18mUsing Dask to Distribute Machine Learning Jobs