On Blocks, Copies and Views: updating pandas' internals
summary
Proposal: any DataFrame or Series derived from another single rule: derived objects behave as copies, so mutating a child never changes the parent. Implementation uses delayed copy-on-write to avoid needless memory churn, with compatibility tooling (warnings, possible transition mode) and a proof-of-concept: an initial prototype implementation exists but requires testing. Benefits: predictable semantics, removal of SettingWithCopyWarning, fewer eager copies. Downsides: breaking behavior for some chained assignments and engineering complexity.
AI-written summary · about
chapters (12)
- 0:00Start
- 2:11Copy vs view
- 3:25Examples: Filtering rows with a boolean mask
- 5:00Examples: subsetting columns
- 5:11Examples: selecting a single column
- 5:22Current situation
- 6:28The Setting with CopyWarning
- 9:50Intermezzo: copies and method chaining
- 12:29Can we do better?
- 14:22Examples: slicing rows
- 17:33Hiding copy/view details with Copy-on-Write
- 20:19Avoiding copies in method chaining
related talks
27mOn copies and views: updating pandas' internals
01h 02mPandas from the inside
01h 17mPandas from the Inside / "Big Pandas"
28mIdiomatic Pandas
24mHow to maximally parallelize the entire pandas API
35mIdiomatic Pandas
25mBuilding a Multi-Core Apply Function for Pandas
37mpandas at a Crossroads, the Past, Present, and Future