How Narwhals brings Polars, DuckDB, PyArrow, & pandas together
summary
Narwhals is a lightweight compatibility layer that makes code accept and return Polars DuckDB PyArrow pandas and cuDF without forcing conversion to pandas, using a simple pattern: from_native → express with Polars-like expressions → to_native. It ships with zero dependencies, a narwhals.stable.v1 compatibility promise, and supports order-aware lazy ops (cumsum via over order by). Plotting users report up to tenfold speedups for Polars versus prior logic, and the project already gathers 100+ contributors and cross-engine SQL and backend support.
AI-written summary · about
chapters (21)
- 0:00Narwhals introduction
- 1:50Life before/after Narwhals
- 4:01Zero dependencies matter
- 6:11Writing agnostic functions
- 10:01Lazy vs eager data frames
- 13:12SQL support in Narwhals
- 14:12Backwards compatibility
- 17:25Community and future plans
- 20:57Q&A: Decorators & use cases
- 22:22Q&A: Performance impact
- 24:56Q&A: Plotly gains
- 26:20Q&A: Framework switching
- 27:23Q&A: Adoption growth
- 28:05Q&A: Why Narwhals name
- 29:33Q&A: SQL translation
- 31:16Q&A: Snowflake & databases
- 32:03Q&A: Future-proofing
- 32:49Q&A: Simulating APIs
- 33:55Q&A: Astropy support
- 34:45Q&A: Fixing pandas bugs
- 35:54Closing
related talks
34mHow you can write a dataframe-agnostic library
30m"Better dataframes"
30mIs the great dataframe showdown finally over? Enter: polars
34mData Wrangling with DuckDB
28mExpressive & fast dataframes in Python with polars
38mExpressive and fast dataframes in Python with polars
35mHow many dataframe libraries do you need to change a lightbulb?
39mPython and SQL: Better Together, Powered by DuckDB