Cleaning, optimizing and windowing pandas with numba
summary
Claims a concrete refactor: separate windowing into indexer, aggregator, kernel so rolling, expanding and ewm reuse kernels and share code; implements a subtractive aggregator to keep running sums and avoid full-window recompute. Uses Cython-style existing code and proposes Numba for JITing user kernels via a Numba JIT class decorator to keep everything in native code and avoid Python callbacks. Work targets maintainability and comparable or better speed, remains under discussion and is not merged into mainstream Pandas.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 1:12Project goals
- 3:44Windowing in finance
- 4:26Rolling windows
- 7:16Window types
- 10:26Pandas history
- 13:40Pandas issues
- 14:25Introduction to scythe on and its role
- 17:20Performance trade-offs
- 20:36Cleaning goals
- 23:14Three-component architecture
- 28:16EEMA support
- 28:58Numba JIT
- 31:29Numba vs scythe
- 33:39Project status
- 35:05Q&A
