Pandas May Be Slow, but Pandas Doesn't Have to Be
summary
Shows concrete wins: converting ints to uint8 and floats to float32 and turning low‑cardinality strings into category shrinks a 286 KB frame to ~17 KB (region 10 KB → 1.3 KB, ~80% savings). Demonstrates slow patterns (index loop 270 ms) versus faster options (collect‑then‑assign, iterrows, apply ≈20 ms, vectorized/numpy ops fastest). Compares multiprocessing (≈50% time), threads (much slower), and pandarallel with parallel_apply (≈2.7 s) and gives the exact code patterns to try.
AI-written summary · about
chapters (8)
- 0:00Start
- 7:24Floats
- 9:41Categoricals
- 14:37Memory Usage
- 22:16Cpu Execution Time
- 28:29Array Split Method
- 30:13Concurrent Futures
- 30:51Pandas Parallel
related talks
29mMaking Pandas Fly
35mIdiomatic Pandas
35mFast Pandas Performance Tricks You Wish You Knew Earlier
01h 17mPandas from the Inside / "Big Pandas"
31mFaster Pandas: Speed Up Your Code, Shrink Your Cloud Bill
28mIdiomatic Pandas
40mBeyond Pandas: The Great Python Dataframe Showdown
01h 02mPandas from the inside