10 years of Parquet- what's next?
summary
Parquet shows concrete wins and clear gaps: a prototype Vortex implementation yields 15% improved compression and roughly 50× faster random access (colleague tweaks reduce that), and Spiral integrates Vortex as a toolkit for compressed Arrow on disk and wire. New codecs discussed include Alp for floats and FSST for strings, and systems like Better Blocks, Lance and Nimble explore external metadata and column‑level layouts. Cloud realities — S3 ~5 ms first byte, machines with 896 cores and 32 TB RAM — drive these design choices.
AI-written summary · about
chapters (17)
- 0:00Parquet introduction and evolution
- 1:09Columnar file formats
- 2:55Analytics workloads and access
- 4:22Hardware shifts in Parquet
- 5:27Parquet file structure and metadata
- 7:33Optimizing footer reads
- 10:20Metadata pruning and statistics
- 12:51Compression and random access
- 14:33Lightweight encodings
- 16:22Better Blocks compression
- 18:08Lossless float compression
- 21:19FSST symbol compression
- 23:31Ten years and future
- 23:52Parquet V3 updates
- 24:55Experimental Parquet projects
- 27:02Vortex file format toolkit
- 28:05Q&A
related talks
08mBackstage interview
37mApache Parquet : the standard, efficient file format for tabular data
46mA 101 in Time Series Analytics with Apache Arrow, Pandas and Parquet
42mColumnar Storage: Redefining Data Management for the Modern Era
43mNew Computer Trends and How This Affects Us
59mPerformance matters - new tricks for old dogs
39mDesigning for high-performance execution with Arrow and Polars
37mPolars 1.0 and beyond