ORC and Parquet: Formats and Their Use on HDFS
summary
Compares the ORC and Parquet columnar file formats for analytics on HDFS, explaining their features, trade-offs, and usage patterns. Covers practical considerations for storing and processing large datasets with these formats on Hadoop ecosystems and when to choose one format over the other.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 1:12Speaker and formats
- 2:59Parquet structure
- 6:17ORC structure
- 11:49Compression comparison
- 16:09Compression benchmarks
- 17:14Small table optimization
- 18:40Sorting and indexes
- 20:50Large table optimization
- 23:23Index importance
- 24:51Bloom filters
- 25:36Format comparison
- 27:29Security features
- 28:40Performance metrics
- 29:23Recommendations
- 29:49Q&A
related talks
37mApache Parquet : the standard, efficient file format for tabular data
46mEfficient Data formats for Analytics with Parquet and Arrow
28mBeyond Parquet and ORC: Upgrading Data Infrastructure for Multi-modal AI with Lance Col...
40mFast Access To Your Complex Data #bbuzz
39mHow to avoid columnar calamities: what no one told you about Apache Parquet
29mEfficient and portable DataFrame storage with Apache Parquet
42mColumnar Storage: Redefining Data Management for the Modern Era
36mTable Formats Change Everything (By Not Changing Anything)