Dask-on-Ray: Using Dask and Ray to Analyze Petabytes of Remote Sensing Data
summary
A Dask-on-Ray scheduler converts each Dask task into a ray.remote call so full Dask graphs run on Ray’s decentralized scheduler and Plasma object store, leveraging zero-copy intranode sharing and Ray fault tolerance. Benchmarks show surprising wins: 2–10× faster on a 96‑core single node, parity on a 5‑node cluster, and 20–50% faster for a 3.5‑year NDVI composite; a previously failing high‑scene composite now completes (thousands of scenes, petabytes touched). This powers a Workflows product that scales to tens of thousands of cores.
AI-written summary · about
chapters (17)
- 0:00Intro
- 0:38Talk Overview
- 3:17Dask's three core components
- 4:29Dask task graph specification
- 4:55Dask task scheduling and execution
- 6:42The pitfalls of Dask Distributed
- 9:13Introduction to Ray
- 10:06Ray Architecture Key Paradigms
- 13:02The Dask on Ray scheduler
- 13:49Dask Ray scheduler usage
- 14:20Dask on Ray scheduler callbacks
- 15:36The groupby benchmark
- 16:18Dask on Ray vs Dask Distributed - Single node groupby
- 17:17Dask on Ray vs Dask Distributed - 5 node cluster
- 19:40Workflows programming model
- 24:23Dask-on Ray backend vs Dask backend benchmarks
- 25:53Future directions for Dask on Ray
related talks
26mData Processing on Ray
27mData Processing on Ray
28mA Deep Dive into Ray’s Scheduling Policy
01h 30mDistributed Python with Ray-Hands on with the Ray 2.0 APIs for scaling Python Workloads
57mIntroduction to Ray for distributed and machine learning applications in Python
35mMars-on-Ray: Accelerating Large-scale Tensor and DataFrame Workloads
39mScaling Python for Machine Learning: Beyond Data Parallelism
21mPowering Open Data Hub with Ray