Skinny Pandas Riding On A Rocket
summary
25 million rows (4 GB CSV) inflate to ~10 GB in pandas memory; converting strings to categorical: 3.4 GB → 219 MB and downcasting numerics cuts RAM and speeds up ops. Installing bottleneck/numexpr triples some numeric ops. Using Dask with parquet partitions processes the dataset in chunks (100 files), resample to weekly completes in ~9 s with ~1.2 GB peak (rises to ~6 GB with 16 workers). SQLite stores a 4 GB single-file DB and answers indexed group-bys in ~2 s. Vaex uses mmap and returns group-bys in <1.5 s with low RAM.
AI-written summary · about
chapters (9)
- 0:00Faster data science workflows
- 4:16Pandas limitations
- 9:05Optimizing pandas
- 13:35Speeding up pandas
- 16:02Dask for larger datasets
- 24:23SQLite as data tool
- 26:30Vagues for big data
- 31:24Choosing the right tool
- 32:26Q&A
related talks
40mBeyond Pandas: The Great Python Dataframe Showdown
31mFaster Pandas: Speed Up Your Code, Shrink Your Cloud Bill
01h 17mPandas from the Inside / "Big Pandas"
26mSmall Big Data: using NumPy and Pandas when your data doesn't fit ...
34mSmall Big Data: using NumPy and Pandas when your data...
14mKeynote: Scalable Python We Can Afford
28mIdiomatic Pandas
29mMaking Pandas Fly