Anatomy of reading Apache Parquet files (from the Apache Impala perspective)
summary
Impala implements a C++ Parquet scanner integrated with its IO and memory manager, not Parquet-mr or parquet-cpp, enabling features like data caching, memory-bounded execution and efficient parallelism. It processes one row group per thread with per-row-group threads, reads columns in separate streams, uses lazy materialization to skip pages, evaluates predicates as LLVM-compiled predicates, supports dictionary/delta encodings and Bloom/min-max page filters, and uses tracked allocations so queries stop safely instead of crashing.
AI-written summary · about
chapters (12)
- 0:00Introduction to talk
- 0:10What is Apache Impala
- 1:56Apache Parquet overview
- 3:00Parquet file format details
- 4:07Dictionary and Delta encoding
- 7:19Compression and filtering techniques
- 10:11Impala Parquet scanner design
- 13:47Memory management in scanner
- 14:49Lazy materialization concept
- 15:31Row group and page filtering
- 19:06Optimizing Parquet files
- 25:09Bloom filter usage
related talks
39mHow to avoid columnar calamities: what no one told you about Apache Parquet
46mEfficient Data formats for Analytics with Parquet and Arrow
40mApache Kudu (incubating): Fast Analytics on Fast Data #bbuzz
31mMapping FHIR data to parquet for analytics and machine learning
22mAccelerating Analytics with Apache Arrow
30mReconciling Kafka, SQL and Parquet
58mHow to transform AVRO (IDL) data to multiple PARQUET files
47mAnalytics at the speed of Direct Lake