5 Steps to Speed Up Your Data-Analysis on a Single Core
summary
Pipeline time drops from multi-day failures to 18 hours then 7 hours, a reported 500–700% improvement and a toy-case speedup of about 200×. Profiling with sampling tool py-spy (vs instrumenting with yappi/line profilers) finds JSON decode and file I/O as hotspots; switching to HDF5 and concatenating NumPy arrays lets group-by via sort/split use vectorized ops. Changing dtypes to 32-bit cuts memory and gives ~20% gain; Numba no‑python compilation adds further wins after the first compile.
AI-written summary · about
chapters (8)
- 0:00Start
- 1:24Large Scale Data Analysis Experiments
- 3:16Why to Speed Up: Example
- 5:28Why to Speed Up on a Single Core
- 13:51Profiling
- 19:35Vectorization
- 20:49Memory & Precision Tradeoffs
- 24:53Further Steps
related talks
30mHow to Tune your Python Analysis Pipeline: A Profiler Guide
30mSpeed Up Your Data Processing
01h 01mParallel Python Analyzing Large Data Sets
26mBodo: Supercomputing-Like Performance and Scale for Python/Pandas
31mFaster Pandas: Speed Up Your Code, Shrink Your Cloud Bill
22mExperiences making CPU-bound tasks run much faster
40mScaling Python - Bank Edition
45mInteractive Analysis of (Large) Financial Data Sets