Pandas on Steroids
summary
A 2.5 GB NYC taxi CSV starts at 51.4 seconds and 6.7 GB memory; dropping unused columns, parsing pickup/dropoff as datetimes and converting borough to category halves memory and speeds things up. Switching CSV reader to the pyarrow engine cuts load from ~17s to 1.3s, switching storage to parquet drops reads to ~157 ms, and replacing row-wise apply with vectorized series operations lowers end-to-end time to ~1.2s. Pandera enforces schemas and catches empty borough strings; enable copy-on-write to avoid setting-with-copy bugs.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 1:05Why Pandas
- 2:30Data
- 2:54Pipeline
- 4:41Reading CSV
- 6:43Using Categories
- 7:39Using Prior Engine
- 9:27Improved Pipeline
- 14:29Problem on Data
- 15:16Tools
- 16:53Conclusion
- 18:13CopyOn
- 19:35Outro
related talks
21mPolars: A highly optimized dataframe library
28mWorking with dates and times in Pandas
28mIdiomatic Pandas
35mFast Pandas Performance Tricks You Wish You Knew Earlier
31mChasing Pandas
35mWorking with data in Python for the Aspiring Data Scientists
25mPandas on Jupyter: Improving Your Data Analysis Workflow
35mIdiomatic Pandas