Making Pandas Fly
summary
Shows concrete wins: converting repeated strings to category dtype halves memory and cuts a value_counts from ~400ms to ~23ms, switching float64 to float32 halves RAM with identical results for coarse stats, float16 saves more but loses precision and can be slower, and calling series.values then numpy reduces work (up to ~10x faster for sum) though pandas + optional *bottleneck library yields ~3x speedups for mean. Includes a small GitHub tool that scans columns for compression opportunities and an IPython memory-usage tracer.
AI-written summary · about
chapters (27)
- 0:00Introduction
- 2:34What we’ll cover in this talk
- 3:23Loading dataset
- 5:11Memory usage with deep flag
- 6:15String column memory
- 7:19RAM usage importance
- 8:23Compressing strings
- 10:09Category dtype performance
- 10:50Index lookups with categories
- 11:55Company ages from datetime
- 13:21Data quality checks
- 14:02Reducing float precision
- 16:49Float16 trade-offs
- 17:32RAM savings recap
- 17:53Dtype diet tool
- 19:05Query methods comparison
- 20:08Speed with numpy values
- 20:49Behind the scenes of pandas operations
- 22:12Pandas vs numpy benchmark
- 23:40Bottleneck speed boost
- 25:01When to use numpy
- 25:42Pandas performance deps
- 26:02RAM tracking with ipython
- 26:2220 years of mistakes
- 27:07Early error testing
- 27:47Efficiency notes repo
- 28:08Summary
related talks
32mPandas May Be Slow, but Pandas Doesn't Have to Be
35mIdiomatic Pandas
31mFaster Pandas: Speed Up Your Code, Shrink Your Cloud Bill
35mFast Pandas Performance Tricks You Wish You Knew Earlier
28mIdiomatic Pandas
40mBeyond Pandas: The Great Python Dataframe Showdown
01h 02mPandas from the inside
01h 17mPandas from the Inside / "Big Pandas"