Advanced Polars: Lazy Queries and Streaming Mode
summary
Polars claims Rust-level speed and Apache Arrow memory layout to run lazy queries that optimize across steps using predicate and projection pushdown and expression simplification, letting a scan + collect handle 54 million+ rows without materializing intermediates. LazyFrames represent the query DAG, optimizers reorder and cache subqueries, and a streaming mode processes data in chunks (configurable chunk size) with engine="streaming" but silently falls back if ops like global sort or cumulative sums force in-memory execution.
AI-written summary · about
chapters (7)
- 0:00Polars beyond generative AI
- 3:39Eager mode limitations
- 8:16Low-level optimizations
- 11:50Lazy query optimization
- 16:34Automatic optimizations
- 19:46Streaming large datasets
- 25:51About the speaker and resources
related talks
30mIs the great dataframe showdown finally over? Enter: polars
49mKeynote on Polars Plugins
38mExpressive and fast dataframes in Python with polars
28mExpressive & fast dataframes in Python with polars
01h 32mAn Introduction to Polars
25mIntegrating Rust with Python...
29mBuild simple & scalable data pipelines with Polars & DeltaLake
45mAccelerating OpenSearch With Streaming: Apache Arrow, Flight, Data...