Spark Backend for Ibis: Seamless Transition Between Pandas...
summary
A PySpark backend for Ibis converts Ibis expression trees into native PySpark DataFrame operations so the same Python Ibis expressions run on both pandas and Spark without code changes. It implements windowing, groupby, UDF and rank translations, demonstrates identical results on pandas and Spark, and keeps execution lazy while deferring optimization to the backend. The project ships a working Spark backend that translates to PySpark code and materializes via execute/compile.
AI-written summary · about
chapters (10)
- 0:00Ibis introduction
- 1:46Pandas to distributed frames
- 8:15Current approach problems
- 11:28Ibis core components
- 15:04Transformations in Ibis
- 17:56Executing Ibis expressions
- 20:05Ibis to PySpark translation
- 26:43Live demo in notebook
- 29:16Key takeaways and future
- 31:25Q&A
related talks
01h 17mIbis: A fast, flexible, and portable tool for data analytics
28mIbis framework, Making data science work at any scale
02h 16mIntroduction to Ibis: blazing fast analytics with DuckDB...
44mIbis- Expressive analytics in Python at any scale
42mIbis: Don't Let the Engine Dictate the Interface
01h 24mBuilding the composable Python data stack with Kedro & Ibis
27mBecause SQL Is Everywhere
37mpandas at a Crossroads, the Past, Present, and Future