Parquet, Go, and Unreasonable Amounts of SIMD
summary
Parquet-go is a production-ready Go library that achieves extreme throughput by implementing core codecs and filters in hand-written assembly using AVX2 and AVX512. Concrete techniques include SIMD bit-packing, delta encoding with validation, byte-slice B-stream split using scatter/gather, and split-block bloom filters that hit roughly 30 GB/s for filter generation and about 1–2 billion hashes/sec for parallel hashing; benchmarks show up to ~60% speedups vs baseline Go implementations.
AI-written summary · about
chapters (6)
- 0:00Introduction to Parquet
- 1:02Performance strategies
- 2:13Encoding and decoding
- 3:40Advanced compression
- 4:23Bloom filter optimization
- 5:31Contributions and conclusion
related talks
31mState of Parquet 2025: Structure, Optimizations, and Recent Innovations
39mHow to avoid columnar calamities: what no one told you about Apache Parquet
37mApache Parquet : the standard, efficient file format for tabular data
29mEfficient and portable DataFrame storage with Apache Parquet
18mPanel: Databases and Hardware
46mEfficient Data formats for Analytics with Parquet and Arrow
26mFast Erasure Coding for Data Storage: A Comprehensive Study of the Acceleration...
28mBuilding a Cutting Edge Search Engine in Go