Extending Pandas using Apache Arrow and Numba
summary
Claim: Pandas gains drop-in custom dtypes by storing data in Apache Arrow buffers and compiling element logic with Numba, yielding a native string implementation called Fletcher that is about 10x smaller than object arrays and shows roughly 2x speed on groupby in examples. It works with Pandas 0.23+ via the ExtensionArray API and is pip-installable, but nullable-int support requires pandas master, first-call JIT compile costs apply, and some operations still fall back to NumPy conversions.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 0:41Speaker's background
- 1:53Introduction to Pandas and NumPy
- 3:05Shortcomings of Pandas
- 8:55Extending Pandas with ExtensionArrays
- 12:16Apache Arrow for Data Storage
- 15:50Numba for computing
- 21:03Putting all together
- 28:10Closing
- 28:45Questions
related talks
24mPolyglot data with python: Introducing Pandas and Apache Arrow
31mThe Arrow revolution in pandas and Dask
29mFaster PySpark with Apache Arrow
33mWhats New In Pandas
41mFaster SQL with pandas and Apache Arrow
31mpandas: Integer NA as a first class citizen
46mA 101 in Time Series Analytics with Apache Arrow, Pandas and Parquet
28mExtending Pandas with Custom Types