Small Big Data: using NumPy and Pandas when your data...
summary
Concrete tactics and numbers: shows how NumPy and Pandas handle data that won’t fit in RAM using compression, chunking, indexing and cheap extra memory, with examples that actually run. Demonstrates dtype downcasting and sparse arrays, zarr-backed arrays and pandas chunked reads, and using Dask for parallel chunks and a SQLite index to avoid scanning. Reports real timings (e.g. max via Dask ~1.1s → 0.3s; indexed query 570ms → 10ms) and explains when each approach reduces memory or speeds up processing.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 2:06Memory limits for big data
- 3:32RAM or cloud solutions
- 5:17Reduce memory usage
- 7:03NumPy data type compression
- 9:11NumPy sparse arrays
- 11:39Pandas data type compression
- 13:03Processing data with CZAR
- 15:53Pandas chunking methods
- 16:55Parallel processing with Dask
- 20:22Indexing relevant data
- 23:10Voter data indexing example
- 28:55Techniques beyond NumPy and Pandas
- 30:17Further resources
related talks
26mSmall Big Data: using NumPy and Pandas when your data doesn't fit ...
01mSmall Big Data: using NumPy and Pandas
01h 17mPandas from the Inside / "Big Pandas"
01h 02mPandas from the inside
01h 22mUsing Dask for Parallel Computing in Python
14mKeynote: Scalable Python We Can Afford
44mUsing Dask for Parallel Computing in Python
35mIdiomatic Pandas