"Better dataframes"
summary
Pandas is now framed as useful but aging, with pandas 2 supports optional Arrow backing as a partial fix. The survey compares concrete tools: Polars and DuckDB neck and neck for local, fast, Arrow-backed analytics (Polars is in-process only; DuckDB can spill to disk), Ibis provides a single API across ~20 backends to scale from local to cloud, and projects like Vaex Modin Dask PySpark Pandas API and Narwhals aim either scale or interoperability. Benchmarks show Polars/DuckDB substantially outperform pandas.
AI-written summary · about
chapters (8)
- 0:00Introduction and background
- 4:20Defining data frames
- 7:28History and role of pandas
- 13:43Pandas future and limitations
- 16:17Alternative libraries
- 21:34Ibis and interoperability
- 24:00Scaling and ecosystems
- 26:46Standardization and Q&A
related talks
35mHow many dataframe libraries do you need to change a lightbulb?
35mA Tour of the Many DataFrame Frameworks
26mLife Beyond Pandas: Workflows with DuckDB, Daft, Polars, and Datafusion
40mBeyond Pandas: The Great Python Dataframe Showdown
36mHow Narwhals brings Polars, DuckDB, PyArrow, & pandas together
32mThe PyArrow revolution in Pandas
30mSpeed is Not All You Need for Data Processing
31mThe Arrow revolution in pandas and Dask