Pandas, Polars and the DataFrame Consortium.
summary
Pandas 2.0 with Arrow is compared to Polars using timed reads, filter/select/unique, aggregations and group+sort on a powerful MacBook with mem profiling; measured results show Polars uses Arrow and Rust and Polars wins speed and memory in tests, plus built-in parallelism and lazy evaluation. Syntax requires pl.col style calls which complicates migrations, and some DB connectors and tooling (including LLM integrations) lack Polars support. The DataFrame Consortium aims to standardize APIs to reduce inconsistent implementations like differing mean functions.
AI-written summary · about
chapters (8)
- 0:00Introduction and agenda
- 1:03Understanding pandas
- 4:29Performance challenges
- 10:11Introducing polars
- 12:47Comparative analysis
- 21:42Standardization and future
- 22:54Conclusion
- 25:19Q&A session
related talks
31mBeyond Pandas: lightning fast in-memory dataframes with Polars
30mIs the great dataframe showdown finally over? Enter: polars
58mFrom Pandas to Polars Upgrading Your Data Workflow
21mPolars: A highly optimized dataframe library
31mPolars, the End of Spark?
29mHow Narwhals is silently bringing pandas, Polars, DuckDB, PyArrow, and more together
15mData Refinement: Unleashing the Power of SQL in Python using Polars
31mChasing Pandas