On copies and views: updating pandas' internals
summary
Proposal simplifies pandas’ copy/view rules so any derived DataFrame or Series behaves as a copy while under‑the‑hood uses views until mutation to save memory. It aims to remove SettingWithCopyWarning, avoid defensive copies from method chains, and delay actual copying until an in‑place mutation occurs. Prototype is available behind an option pd.options.mode.copy_on_write in pandas 1.5+ so users can try the change today.
AI-written summary · about
chapters (15)
- 0:00pandas internals intro
- 1:11Understanding the SettingWithCopyWarning
- 2:35Copies vs views
- 4:03Mutations affect originals
- 6:08Copy behavior problems
- 7:34Chained assignments pitfalls
- 9:46Defensive copying
- 11:10Method chains copies
- 12:55Simplified copy rules proposal
- 15:44New copy rules consistency
- 17:09Eliminating the SettingWithCopyWarning
- 18:54Copy-on-write explained
- 20:43Optimizing method chains
- 21:05Try new behavior
- 22:33Summary and feedback
related talks
30mOn Blocks, Copies and Views: updating pandas' internals
01h 02mPandas from the inside
25mBuilding a Multi-Core Apply Function for Pandas
01h 17mPandas from the Inside / "Big Pandas"
37mpandas at a Crossroads, the Past, Present, and Future
28mIdiomatic Pandas
42mDemystifying pandas internals
35mIdiomatic Pandas