Give me the Data
summary
CSV works reliably for small files and long-term reproducibility but hits RAM limits once datasets approach gigabytes, so use readr or data.table for tidy input. Parquet with columnar layout, dictionary encoding and compression (example: a large frame shrinks ~10% vs RDS; write speed: RDS 27s, Parquet snappy 4s, gzip 12s) speeds I/O and reduces storage when paired with Parquet + Apache Arrow. For repeated queries or joins load into DuckDB as caching engine and execute fast on-disk analytics.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 3:13Data to results in 25 years
- 3:55Reading small data with CSV
- 6:24In-memory data limits
- 7:05Why Parquet is a better file format
- 9:56Row vs column storage
- 10:38Parquet compression
- 12:25Arrow for large files
- 13:28RDS vs Parquet performance
- 14:50Format reading speeds
- 15:56DuckDB for big data
- 17:26Choosing the right tool for your problem
- 18:07Q&A
related talks
31mData wrangling like a boss with Duckdb
37mDuckDB: Crunching Data Anywhere, From Laptops to Servers
29mEfficient and portable DataFrame storage with Apache Parquet
42mColumnar Storage: Redefining Data Management for the Modern Era
40mFast Access To Your Complex Data #bbuzz
22mAccelerating Analytics with Apache Arrow
01h 02mAnalytics for not-so-big data with DuckDB
46mEfficient Data formats for Analytics with Parquet and Arrow