Pandas from the Inside / "Big Pandas"
summary
Pandas internals get examined: DataFrames are blocks of homogeneous numpy arrays managed by a block manager, so pulling a Series is often microsecond-fast while naïve DataFrame ops are tens to hundreds of times slower — one example shows 70× faster for column access and >100× differences for some sums. A 14,000-row example becomes real-world when CSVs of 450k rows hit 2.3 GB; switching to parquet plus Dask lets a staged lazy query show results in milliseconds but requires a final Dask compute step that takes seconds.
AI-written summary · about
chapters (31)
- 0:00Pandas internals intro
- 3:20Pandas memory efficiency
- 12:24Numpy foundations
- 14:34Python and numpy combo
- 20:50Pandas analysis workflow
- 21:33Football data example
- 26:58Sports data loading
- 29:54Column extraction speed
- 30:37Pandas block manager
- 39:18Indexing and data types
- 40:45Reformatting data
- 45:18Multi-index selection
- 48:17Avoid row-by-row ops
- 49:07Adding new columns
- 51:16Consolidation in pandas
- 53:10Group by performance
- 55:22Percentages and sorting
- 56:25Efficient pandas summary
- 1:00:08Distributed pandas intro
- 1:02:57Scaling to big data
- 1:03:19Dask as pandas replacement
- 1:04:45Flight data with dask
- 1:06:15CSV inefficiency
- 1:07:20Dask flight performance
- 1:08:25Dask task graphs
- 1:10:07Parquet for storage
- 1:11:12Flight delays visualization
- 1:12:17Monitoring calculations
- 1:13:27Dask internals
- 1:15:36Dask limitations
- 1:16:00Conclusion
related talks
01h 02mPandas from the inside
01h 21mPandas from the Inside
42mDemystifying pandas internals
32mPandas May Be Slow, but Pandas Doesn't Have to Be
40mBeyond Pandas: The Great Python Dataframe Showdown
34mSmall Big Data: using NumPy and Pandas when your data...
35mFast Pandas Performance Tricks You Wish You Knew Earlier
35mIdiomatic Pandas