The Column's the limit
summary
Buckaroo is a concrete, open‑source notebook table that actually reads big files without filling RAM: it uses viewport streaming, lazy Polars pipelines, batched background stats, a series cache and a tiny Rust extension for column-level hashing so warm loads render the first viewport under a second. The demo opens a 19M‑row Parquet (≈450 MB), survives up to ~100M rows on a commodity laptop by throttling UI updates and running background tasks with timeouts, and exposes processed pandas frames for saving or further work.
AI-written summary · about
chapters (10)
- 0:00Buckaroo and Polars intro
- 1:08Buckaroo live demo large data
- 4:00Why Buckaroo for data exploration
- 8:56Extending Buckaroo analytics
- 14:20Buckaroo large data approach
- 19:03Buckaroo efficient processing
- 25:26Polars integration lazy eval
- 28:22Debugging dataframes
- 30:56Summary
- 32:00Q&A
related talks
42mColumnar Storage + Python: Powering Modern Data Science and Analytics
34mClickHouse® - What is Behind the Fastest Columnar Database?
20mGive me the Data
26mColumnar Storage: Redefining Data Management for the Modern Era
29mEfficient and portable DataFrame storage with Apache Parquet
37mDuckDB: Crunching Data Anywhere, From Laptops to Servers
40mDuckDB: Harnessing in-process analytics for data science and beyond
32mDataFrames: scaling up and out