Building a Multi-Core Apply Function for Pandas
summary
This talk demonstrates building a multi-core apply method for pandas to speed up slow DataFrame.apply calls without large memory overhead. It covers Python multiprocessing techniques (including invoking copy-on-write memory) and pandas internals—how DataFrames are structured—and shows code examples and a simple pandas extension that implements the parallel apply.
AI-written summary · about
chapters (14)
- 0:00Multi-core apply in pandas
- 1:29Processes and multi-core basics
- 3:11Memory sharing challenges
- 5:39Pandas data frames
- 7:27Memory optimization strategies
- 9:13Global memory access
- 11:01Monitoring memory usage
- 13:08Virtual memory and isolation
- 15:37Process spawning and mapping
- 17:23Data frame structure
- 19:11Why pandas objects copy
- 21:16Pandas multi-core extensions
- 22:20Memory-efficient apply
- 23:22Performance benefits
related talks
01h 02mPandas from the inside
24mHow to maximally parallelize the entire pandas API
44mHigh Perfomance Pandas
30mSpeed Up Your Data Processing
26mBodo: Supercomputing-Like Performance and Scale for Python/Pandas
32mPandas May Be Slow, but Pandas Doesn't Have to Be
35mFast Pandas Performance Tricks You Wish You Knew Earlier
01h 56mParallelizing Scientific Python with Dask