The Secret Life Of Rolling Pandas
summary
Shows how pandas and related libraries implement fast rolling aggregations and what actually gets built and why: running sums via cumulative-sum trick for O(1) per window, O(n) sliding-sum updates, inclusion–exclusion for 2D windows, numeric pitfalls that produce negative variances and how Welford's online algorithm fixes them, min/max via a deque (monotonic queue), and median via a randomized skiplist-like structure with average-case guarantees. Mentions bottleneck → pandas code, memory vs speed tradeoffs, and which approaches fail or succeed in practice.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 1:24Study rolling pandas
- 3:10Open-source call
- 3:32What rolling pandas are
- 5:24Climate data aggregations
- 8:44Rolling window mechanics
- 12:58Efficient rolling sum
- 16:17Inclusion-exclusion principle
- 18:52Rolling variance without errors
- 22:03Welford’s algorithm
- 23:10Rolling minimum and maximum with dequeues
- 26:04Rolling median variants
- 30:04Q&A
