How to incrementally scale existing workflows on Spark, Dask or Ray?
summary
Shows a concrete path to scale pandas workloads by layering Fugue over Spark Dask or Ray, using a transform wrapper and cost-partitioning so existing Python functions run unchanged on clusters. Demonstrates DuckDB for local iteration, shows native Spark SQL on Photon can be worse, and reports a real case with about 10x speedup from the partitioning approach. Migration is additive, reversible, platform-agnostic, and keeps unit tests working.
AI-written summary · about
chapters (8)
- 0:00Incremental scaling on Spark/Dask/Ray
- 1:02Incremental adoption of distributed computing
- 4:43Migration challenges and myths
- 10:48Fugue for incremental adoption
- 17:15Step-by-step incremental code
- 21:51Incremental SQL with Fugue
- 30:52Balancing value and effort
- 32:20Q&A
related talks
31mComparing the Different Ways to Scale Python and Pandas Code
39mScaling Python for Machine Learning: Beyond Data Parallelism
25mScale Data Science by Pandas API on Spark
28mAdapting from Spark to Dask: what to expect
01h 17mScaling PyData with Dask
43mPySpark in Practice
25mDistributed data workflows: PySpark vs Dask
01h 30mDistributed Python with Ray-Hands on with the Ray 2.0 APIs for scaling Python Workloads