Bridging Pandas and Polars
summary
Benchmarks show conversion cost is concrete not theoretical: numeric round trips cost about 250–300 ms conversion and Parquet reads into Pandas take ~3 s while Polars reads run ~300 ms. String-heavy frames explode to 5–8 s for string conversion unless Arrow layouts are used. A database that implements ArcticDB double native integration and direct Arrow/NumPy encoders avoids those copies and can return a 100M-row time‑indexed range in ~1 ms range query, so the built solution works and pays off.
AI-written summary · about
chapters (11)
- 0:00Data frame interoperability
- 1:27Why interoperability matters
- 4:38Memory layout differences
- 10:00Hidden conversion costs
- 13:14ArcticDB integration
- 14:19Benchmarking conversions
- 17:51Sparse data handling
- 22:53Schema conflicts
- 26:20Pandas and Polars indices
- 30:36Time series indexing
- 32:23Key takeaways
related talks
01h 22mAn Introduction to Pandas 2, Polars, and DuckDB
31mPolars on Spark
32mThe PyArrow revolution in Pandas
25mSpeed Up Data Access with PyArrow Apache Arrow Data is the new API
39mWhat we learned by converting a large codebase from Pandas to Polars
30mIs the great dataframe showdown finally over? Enter: polars
46mA 101 in Time Series Analytics with Apache Arrow, Pandas and Parquet
28mWhy you should stop pretending your sparse data is dense