Using Row Groups for fast filtering of large parquet files
summary
Shows that tuning Parquet row groups and layout cuts IO and cloud cost: reading footer metadata with pyarrow’s ParquetFile and adding per‑group stats (min/max, sums, custom version tags) lets engines push down filters via DuckDB query plan checks. On a 3 GB dataset (18M rows) splitting into 500,000‑row row groups moves a full‑scan case from ~34s to roughly 40–50× speedup; sorting by key further yields ~50× gains and far fewer s3fs read calls.
AI-written summary · about
chapters (6)
- 0:00Introduction to Parquet
- 10:22Understanding row groups
- 17:37Benchmarking row groups
- 21:31Optimizing data organization
- 24:49Choosing row group size
- 27:03Cost savings with row groups
related talks
29mEfficient and portable DataFrame storage with Apache Parquet
37mApache Parquet : the standard, efficient file format for tabular data
22mAccelerating Analytics with Apache Arrow
32mConnecting PyData to other Big Data Landscapes using Arrow and Parquet
46mEfficient Data formats for Analytics with Parquet and Arrow
39mHow to avoid columnar calamities: what no one told you about Apache Parquet
20mGive me the Data
31mData wrangling like a boss with Duckdb