Junior vs panda: anti-patterns of the pandas library
summary
Presents common anti-patterns when using the pandas library for data processing, identifies specific inefficient usages, and shows concrete faster alternatives (benchmarked) that improved real-project code performance up to ~60× and made results more stable. Covers benchmarks comparing approaches, selected tips from blog-style sources, and a case study about rewriting a junior-developed pandas job into a more reliable, faster pipeline.
AI-written summary · about
chapters (22)
- 0:00Introduction
- 1:44Pandas performance issues
- 3:07Project constraints
- 4:31DataFrame iteration: itertuples vs iters
- 7:20Iteration test results
- 8:03Apply method optimization
- 10:51Apply with raw argument
- 11:33Merge operations and alternatives
- 14:02Reindex for optimization
- 15:04Merge vs join comparison
- 16:09Merge with indicator
- 17:33Merge optimization results
- 17:53Data type optimization
- 21:02int8 vs int64 types
- 21:43Categorical and sparse types
- 24:54Data type replacement results
- 25:15Code parallelization
- 26:39Using pandarallel
- 27:00Final optimization results
- 27:22Recommendations and conclusions
- 29:50Documentation and benchmarks
- 30:32Q&A
related talks
32mPandas May Be Slow, but Pandas Doesn't Have to Be
35mIdiomatic Pandas
42mDemystifying pandas internals
01h 48mPandas for Beginners
37mpandas at a Crossroads, the Past, Present, and Future
28mIdiomatic Pandas
01h 17mPandas from the Inside / "Big Pandas"
40mBeyond Pandas: The Great Python Dataframe Showdown