Benefits of columnar data layout
summary
Columnar layout cuts disk I/O and memory use by storing each field together, enabling Google Dremel’s definition/repetition level scheme to skip nulls and rebuild rows lazily, and enabling bit-packing, delta encoding and dictionary encoding that can shrink storage an order of magnitude in practice. File formats like Apache Parquet implement these encodings (fast reads, predicate and block pruning) while Apache Arrow brings the same layout to memory for zero‑copy, cross‑language access and SIMD acceleration; tradeoff: writes and row materialization cost more.
AI-written summary · about
chapters (6)
- 0:00Introduction and background
- 1:13History of columnar storage
- 8:26Row vs columnar storage
- 14:34Apache Parquet deep dive
- 23:26Apache Arrow in-memory format
- 32:00Key takeaways and Q&A
related talks
26mColumnar Storage: Redefining Data Management for the Modern Era
39mHow to avoid columnar calamities: what no one told you about Apache Parquet
46mEfficient Data formats for Analytics with Parquet and Arrow
20mGive me the Data
02mLightning Talk - Parquet: The Pillar of Modern Telemetry...
36mTable Formats Change Everything (By Not Changing Anything)
48mDesign of data-intensive applications
39mColumnar Table Performance Enhancements Of Greenplum Database