Multi Dimensional Clustering with Z Ordering
summary
Shows that multi-dimensional file skipping is practical by computing a per-row Z-Ordering value (by interleaving bits of lexicographically encoded columns) and storing file-level z min/max to prune files. Implementation prototypes as an Apache Iceberg rewrite-data-files option that sorts by z-value instead of hierarchical columns. Signed ints need sign-byte flipping, floats and strings need bounded encodings (strings often truncated). Result cuts files scanned dramatically (examples go from scanning many files down to one or two); work is prototyped and integrates with Iceberg metrics.
AI-written summary · about
chapters (13)
- 0:00Intro
- 1:40The Biggest Problem in Big Data is the Data
- 2:42No Sorting - No Partitioning
- 5:31Sorting is Even Better: Selective metrics let us avoid files
- 8:01Enter Space Filling Curves
- 8:26Let's Draw a Trivial Line
- 11:37Z-Order Curve
- 13:25Hilbert Curve
- 16:16Can we use Z-Values in Planning?
- 19:29How do we Compute Curve Values?
- 21:18Getting Lexicographically Orderable Representations
- 23:09Implementation as Rewrite Datafile with Sort
- 24:04Lots of Work Left - Join us! Apache Iceberg is an Open Source Project
related talks
52mApache Iceberg - An Architectural Look Under the Covers
38mMultidimensional Indexing and Sampling on the Lake
32mBeyond Zstandard: The Future of Data Compression
42mYou Migrated to Apache Iceberg—Now What? Speed, Efficiency, and Cost in the Real World
37mApache Iceberg: An Architectural Look Under the Covers
01h 04mAn Extremely Technical Overview of How Apache Iceberg Planning Actually Works
41mTip of the Iceberg
19mA zone-aware cache system for distributed databases