Ibis: Don't Let the Engine Dictate the Interface
summary
Ibis is a pure‑Python dataframe interface that hands queries to engines instead of running them locally, fetching only table schemas until execution and compiling expressions to the engine’s SQL. It supports 20+ engines (DuckDB, Snowflake, BigQuery, Polars, pandas, etc.), reads Parquet/Arrow, and offers deferred execution. Example numbers: a 22M‑row aggregation runs ~1.75s vs 72s (Polars vs pandas), and a 1.1B‑row run finishes on a laptop with ~1.1B rows on laptop memory under 10GB.
AI-written summary · about
chapters (10)
- 0:00Ibis introduction
- 1:05What Ibis is and why it matters
- 3:11Decoupling interfaces from engines
- 7:24Deferred execution with remote data
- 13:26Ibis with DuckDB
- 20:51SQL dialects problem and Ibis solution
- 22:39Performance across engines
- 28:58Real-world example with 1.1 billion rows
- 31:07Closing thoughts and future-proofing
- 33:11Q&A
related talks
01h 17mIbis: A fast, flexible, and portable tool for data analytics
02h 16mIntroduction to Ibis: blazing fast analytics with DuckDB...
28mIbis framework, Making data science work at any scale
44mIbis- Expressive analytics in Python at any scale
27mBecause SQL Is Everywhere
37mSpark Backend for Ibis: Seamless Transition Between Pandas...
39mPython and SQL: Better Together, Powered by DuckDB
32mBecause SQL is everywhere but you don't want to use it