Pandas from the inside
summary
Shows how DataFrames are implemented: data lives in numpy-backed block manager chunks, not as boxed Python objects, so single-column access is cheap — selecting a Series is ~2 μs — while building new DataFrames from columns costs much more. Demonstrates concrete memory savings (thousands vs tens of thousands of bytes), the pitfall of view vs copy and the strange setting-with-copy warning, and a practical speed contrast for aggregation — groupby+concat 200 ms vs 4 ms — plus a brief note that Dask solves out-of-memory distribution.
AI-written summary · about
chapters (22)
- 0:00Introduction
- 1:10Talk goals and pandas overview
- 3:00Python vs NumPy memory and performance
- 8:28NumPy two-dimensional arrays
- 12:24Pandas origins and NumPy
- 15:37Pandas data structures and complexity
- 20:43Typical pandas workflow
- 26:04Loading data into DataFrame
- 27:49DataFrame structure
- 31:27Selecting DataFrame columns
- 36:28Block managers and caching
- 38:41Column selection internals
- 41:53Reformatting columns and indexes
- 44:25Multi-indexes display
- 45:09How indexes work
- 46:38Index alignment in operations
- 49:31Efficient row subset selection
- 51:38Adding columns and warnings
- 54:05Calculating wins, draws, losses
- 55:31GroupBy subtotaling
- 57:45Conclusions and performance
- 59:48Q&A
related talks
01h 17mPandas from the Inside / "Big Pandas"
01h 21mPandas from the Inside
34mSmall Big Data: using NumPy and Pandas when your data...
40mBeyond Pandas: The Great Python Dataframe Showdown
32mPandas May Be Slow, but Pandas Doesn't Have to Be
29mMaking Pandas Fly
01h 04mVisualising Data through Pandas
44mHigh Perfomance Pandas