duckplyr: Analyze large data with full dplyr compatibility
summary
duckplyr 1.0.0 on CRAN lets existing dplyr pipelines run against DuckDB-backed Parquet and remote HuggingFace datasets with no SQL translation and automatic ALTREP lazy vectors so you usually don’t need to call collect. Examples show one million rows grouped into 40k groups finishing in 5 seconds, and a targeted change cuts a 30s pipeline to 10s. It enforces prudence protection against pulling huge datasets, supports as_duckdb_tibble for explicit control, and exposes DuckDB funcs via dd:: and extensions like stochastic.
AI-written summary · about
chapters (16)
- 0:00duckplyr introduction
- 1:10Why use DuckDB and R
- 2:36duckplyr vs dbplyr
- 3:40Access DuckDB in R
- 5:11Demo: HuggingFace remote data
- 6:56Large data in ggplot
- 8:02Local vs remote speed
- 8:46Optimize slow dplyr pipelines
- 9:30Faster dplyr code strategies
- 10:14Best effort mode limits
- 11:02Optimize with as_duckdb_tibble
- 12:27Check results consistency
- 13:10DuckDB functions with dd::
- 13:33Load missing function extensions
- 14:38Nested data in duckplyr
- 15:20Q&A
related talks
58mData Wrangling [for Python or R] Like a Boss With DuckDB
28mAnalyze data easily with duckdb - and the implications on data architectures
45mAnalytics for not so big data with DuckDB
06mA quack at building scalable data pipelines with DuckDB
25mA quack of all trades: DuckDB, a versatile analytical database to keep in your toolkit
37mDuckDB: Crunching Data Anywhere, From Laptops to Servers
38mFrom Big Data to DuckDB: Ryan Boyd (MotherDuck) on Hyper-Tenancy, Duck Lake, and Making Data Fun.
51mMotherduck: Data Analytics in the Post Big-Data Era