The PyData map: Presenting a map of the landscape of PyData tools
summary
A practical map of the PyData landscape that names tools and shows where each actually helps: pure Python and toolz for small fast work, NumPy/Pandas for in‑memory analysis, Blaze to write expressions then execute them on Postgres or other back ends, bcolz compresses 1GB to 51MB (~7×) in a ctable, Dask handles medium data (examples: ~342k UK/Ireland schools found; come into play ≈16GB+), Spark for massive distributed workloads. 1GB→51MB bcolz (≈7×) Dask finds ~342k schools; medium data >RAM (≈16GB) Blaze: write expression, execute on Postgres
AI-written summary · about
chapters (22)
- 0:00PyData landscape intro
- 1:50Why Python for analytics
- 3:56NumPy and pandas updates
- 4:37Python 2.6 data problem
- 6:43Functional programming with PyTools
- 7:47Pandas and NumPy roles
- 8:31XArray for labeled data
- 9:37Blaze decoupling expressions
- 10:40Dask for parallel computing
- 13:27Bcolz fast storage
- 15:55Quantopian’s Bcolz case
- 17:39Dask arrays for big data
- 19:26When to use Dask
- 20:50Speeding up Python with Numba
- 21:55Python big data with Dask
- 23:42Arrow for data formats
- 25:06Ibis SQL-like analytics
- 26:31Using the PyData map
- 27:56Stats and ML in PyData
- 28:58Statsmodels and scikit-learn
- 29:41Future of PyData tools
- 30:22Call to action
related talks
34mSmall Big Data: using NumPy and Pandas when your data...
43mKEYNOTE: Scaling Out PyData
50mBig Data Processing and Data Science Environment in Python
01h 55mAll Them Data Engines: Data Munging with Python circa 2023
37mThe Python Ecosystem for Data Science: A Guided Tour
41mKeynote: Making the Big Data ecosystem work together with Python
01h 01mParallel Python Analyzing Large Data Sets
40mBeyond Pandas: The Great Python Dataframe Showdown