Single node shared memory comes to dask
summary
A Dask fork copies pickled buffers into shared memory so processes map the same bytes, demonstrating that scatter/replicate reduces memory from six workers to one copy instead of 6× duplication. Benchmarks use Arrow LMDB Vineyard backends; LMDB is marginally slower. Shared memory works well for explicit replication but gives no speedup for typical dataframe/array shuffles. Important failures: Dask’s scheduler and memory accounting remain unaware of shared segments, and pickling sliced views forces new buffers so accidental copies still occur.
AI-written summary · about
chapters (10)
- 0:00Dask and shared memory intro
- 2:13Dask distributed clusters
- 4:24Dask cluster deployment
- 7:34Memory errors in Dask
- 9:38Single-node Dask benefits
- 12:28Shared memory vs TCP
- 14:36Shared memory in Dask
- 17:05Shared memory benchmarks
- 19:13Shared memory limitations
- 22:42Summary and future
related talks
30mPandas + Dask DataFrame 2.0 - Comparison to Spark, DuckDB and Polars
28mAdapting from Spark to Dask: what to expect
44mWorking Around the GIL with asyncio
01h 22mUsing Dask for Parallel Computing in Python
01h 03mDask: Next Steps in Parallel Python
18mUsing Dask to Distribute Machine Learning Jobs
01h 26mDask tutorial
31mThe Arrow revolution in pandas and Dask