Scaling PyData with Dask
summary
Hands-on tutorial shows how to scale Pandas and scikit-learn using Dask on a cloud Jupyter/Kubernetes cluster, actually launching a cluster with 20 workers, 2 cores each, 140 GB and reading CSVs from Google Cloud into Dask dataframes and delayed workflows split into 365 partitions. It reports aggregating about 245 million rows, demonstrates persisting to memory, index-based speedups (time-series resample), delayed parallelism cutting multi-step sleeps from 3s to ~2s, and the dashboard and adaptive scaling working live.
AI-written summary · about
chapters (9)
- 0:00Setup for Dask tutorial
- 1:30Introduction to Dask
- 10:13Scaling pandas with Dask DataFrames
- 22:54Optimizing Dask DataFrame performance
- 33:50Dask collections beyond DataFrames
- 41:09Parallelizing workflows with Dask Delayed
- 54:05Distributed scheduling with Dask Delayed
- 1:03:22Machine learning with Dask ML
- 1:07:49Q&A
related talks
01h 23mData Science at Scale with Dask
38mScalable Machine Learning with Dask
01h 03mDask: Next Steps in Parallel Python
25mDeploying Python at Scale with Dask
01h 22mUsing Dask for Parallel Computing in Python
29mScaling Data Workloads with Cluster Managers
01h 26mDask tutorial
01h 56mParallelizing Scientific Python with Dask