State of Parquet 2025: Structure, Optimizations, and Recent Innovations
summary
Format details: file layout with row groups, column chunks, pages and footer; page indexes and Bloom filters let readers skip pages by min/max and probabilistic checks; encodings include dictionary, RLE, delta byte array and common compressors like Zstandard; columnar encryption uses AES-GCM/CTR (128/192/256) but key management sits in the application. Recent innovations: content defined chunking claims ~10% storage savings on real training sets; geometry and geography types land in 2025 enabling efficient geospatial storage; variant type allows partial shredding of binary blobs for columnar queries.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 3:16Parquet standard
- 7:52File structure and metadata
- 11:31Schema design
- 14:23Encoding and compression
- 18:38Columnar encryption
- 20:49Page indexes and bloom filters
- 23:21GeoParquet support
- 24:49Chunking for updates
- 25:33Variant type
- 26:37Limitations and future
- 27:41Q&A
related talks
37mApache Parquet : the standard, efficient file format for tabular data
39mHow to avoid columnar calamities: what no one told you about Apache Parquet
29mEfficient and portable DataFrame storage with Apache Parquet
46mEfficient Data formats for Analytics with Parquet and Arrow
28mBeyond Parquet and ORC: Upgrading Data Infrastructure for Multi-modal AI with Lance Col...
42mColumnar Storage: Redefining Data Management for the Modern Era
39mORC and Parquet: Formats and Their Use on HDFS
30mUsing Row Groups for fast filtering of large parquet files