Pandas on DuckDB with Ponder
summary
Ponder translates pandas to SQL and pushes computations down to your warehouse, producing large generated queries (example: 300+ lines for a min‑max normalization) so operations like normalization and one‑hot encoding run where the data lives. It supports numpy preliminarily, integrates with Modin as an orchestrator, and runs on DuckDB, BigQuery, Snowflake. Reported end‑to‑end speedups on DuckDB are 4–10× and it’s available via pip and a hosted signup.
AI-written summary · about
chapters (6)
- 0:00Introduction to Ponder
- 0:19Pandas popularity and usage
- 2:02Limitations of pandas
- 2:46The Modin project
- 3:25Ponder's SQL translation
- 4:15Using Ponder for scaling
related talks
34mData Wrangling with DuckDB
27mIn Process Analytical Data Management with DuckDB
39mPython and SQL: Better Together, Powered by DuckDB
28mAnalyze data easily with duckdb - and the implications on data architectures
30mScaling Outside the Warehouse Using DuckDB and Python
25mProcessing Biggish Data with DuckDB and Python
24mpg_duckdb: Ducking awesome analytics in Postgres
26mLife Beyond Pandas: Workflows with DuckDB, Daft, Polars, and Datafusion