Working Around the GIL with asyncio
summary
Concrete claim: using asyncio plus OS shared memory avoids pickle serialization overhead and yields a ~25× speedup on a 45 GB synthetic dataset (about 2 billion records), cuts peak RAM roughly fivefold, and reduces per-worker startup latency from many seconds to milliseconds. Implementation uses Python 3.11, multiprocessing.shared_memory with numpy memoryviews, a ProcessPoolExecutor orchestrated by asyncio task groups, and task/exception groups to catch parallel failures; the prototype works on macOS and WSL.
AI-written summary · about
chapters (4)
- 0:00Start
- 3:33MAP / REDUCE
- 31:0125X IMPROVEMENT
- 33:16no serialization overhead -true parallel access to shared data
related talks
31mThinking Outside the GIL with AsyncIO and Multiprocessing
30mParallel computing in Python: Current state and recent advances
30mSpeed Up Your Data Processing
30mPython Serialization and the Hallows of Parallel Computing
26mSingle node shared memory comes to dask
30mOvercoming GIL with subinterpreters and immutability
38mDemystifying the GIL
34mUnderstanding of distributed processing in Python