Scalable Machine Learning with Dask
summary
Dask scales NumPy, pandas and scikit-learn onto a cluster so code stays familiar while data and compute move out of core; examples include a cluster of 100 workers running a random forest where increasing from 100→1000 trees drops single‑machine 120s fits toward distributed time (12s single‑machine then overheads), a custom k‑means_parallel that changes initialization to work distributed, and a Hyperband implementation for large‑data hyperparameter search; transient worker failures appear but the scheduler reschedules tasks and workflows complete.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 1:02Why Dask exists
- 3:59Dask architecture
- 5:26CPU vs memory ML
- 7:13Distributed scikit-learn
- 8:34Demo: Random forest
- 14:26Hyperparameter search
- 20:58Memory-bound solutions
- 24:33Parallel inference
- 25:34Demo: K-means
- 30:36Hyperband tuning
- 33:50Dask ML tools
- 34:33Use cases recap
- 35:28Q&A
related talks
01h 17mScaling PyData with Dask
01h 23mData Science at Scale with Dask
18mUsing Dask to Distribute Machine Learning Jobs
25mDeploying Python at Scale with Dask
01h 03mDask: Next Steps in Parallel Python
01h 22mUsing Dask for Parallel Computing in Python
23mData Science at Scale Using dask and Numba
01h 56mParallelizing Scientific Python with Dask