How to avoid columnar calamities: what no one told you about Apache Parquet
summary
Parquet wins size and speed by columnar layout, run-length and dictionary encodings, and predicate pushdown; a cited example compresses 50M synthetic payments from CSV 2.5GB to Parquet <500MB. Interop pitfalls appear when JVM pipelines write Parquet and Python reads it: common readers (pandas via pyarrow) materialize dictionary encoding to Python objects, inflating memory and losing columnar benefits. Workarounds use the Arrow API to preserve dictionary metadata so Pandas gets categoricals, and note parquet-tools is deprecated so inspect metadata via pyarrow notebooks.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 2:07Agenda
- 3:11Why commoner formats
- 6:50Memory hierarchy
- 10:54Main memory
- 15:25Row oriented representation
- 17:31Parquet is ubiquitous
- 20:03Parquet is more efficient
- 24:17Parquet Tools
- 25:48Parquet Myths
- 27:47Demo
- 33:24Conclusions
related talks
29mEfficient and portable DataFrame storage with Apache Parquet
37mApache Parquet : the standard, efficient file format for tabular data
32mConnecting PyData to other Big Data Landscapes using Arrow and Parquet
46mEfficient Data formats for Analytics with Parquet and Arrow
42mColumnar Storage: Redefining Data Management for the Modern Era
02mLightning Talk - Parquet: The Pillar of Modern Telemetry...
33mFree Your Esoteric Data Using Apache Arrow and Python
26mAnatomy of reading Apache Parquet files (from the Apache Impala perspective)