Strategies for Working with Data as It Grows
summary
Practical techniques to reduce memory and handle growing datasets using Python tools are presented. It covers concrete memory-saving transformations (type changes, downcasting, sparse and categorical encodings, chunked reads) and demonstrates using Pandas, Modin for multi-core drop-in scaling, Vaex (mmap HDF5/parquet) to work with ~40 GB on 8 GB RAM, and PySpark for terabyte-scale processing on a cluster.
AI-written summary · about
chapters (6)
- 0:00Introduction
- 3:33The Problem
- 5:40Getting to Know Pandas
- 13:32Getting to Know Modding
- 22:16Getting to Know Spark
- 26:24Final Notes
related talks
26mSmall Big Data: using NumPy and Pandas when your data doesn't fit ...
32mDataFrames: scaling up and out
01h 03mSqueeze (gently) your data
53mTips and tricks for robust big data applications
35mData Processing Pipelines for Small Big Data
50mBig Data Processing and Data Science Environment in Python
03h 07mData of Unusual Size: Interactive Visualization
30mRetooling for a Smaller Data Era