pandas at a Crossroads, the Past, Present, and Future
summary
pandas sits at a concrete crossroads: maintainers choose between a risky full rewrite, steady incremental improvement, or embracing an API layer (Ibis) that dispatches to Arrow/Acero and Dask/Spark backends. The library has 80 million+ monthly downloads and powers nearly every Fortune 100 and ~90% of Two Sigma research workflows, so compatibility matters. Practical moves include dispatch to Arrow, Numba and multithreaded readers, and Two Sigma’s Ibis-based stack that lets identical code run on pandas Spark or a streaming Acero backend — an approach that already works in practice.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 2:11What is pandas
- 4:40Pandas in Python ecosystem
- 6:48Pandas history and funding
- 10:43Maintaining pandas and API
- 13:34Pandas-like libraries
- 17:06Pandas challenges and limits
- 21:40Pandas future options
- 24:47Pandas expressiveness at Two Sigma
- 28:40Two Sigma scaling pandas
- 29:43Summary
- 30:48Q&A
related talks
25mScale Data Science by Pandas API on Spark
33mWhats New In Pandas
30mIs the great dataframe showdown finally over? Enter: polars
30mPandas + Dask DataFrame 2.0 - Comparison to Spark, DuckDB and Polars
14mKeynote: Scalable Python We Can Afford
40mBeyond Pandas: The Great Python Dataframe Showdown
01h 17mPandas from the Inside / "Big Pandas"
24mPolyglot data with python: Introducing Pandas and Apache Arrow