Multidimensional Indexing and Sampling on the Lake
summary
A multi‑dimensional index (Cubist / Qbeast format) stores one file per spatial cube with per‑file weights so queries can read only matching files, run sampling pushdown to stop once a target fraction is reached, and return fast approximate aggregates; internal benchmarks report 95% of queries faster. Implementation targets Delta Lake/Iceberg on Spark, compaction and timestamp indexing are planned, v0.3.0 supports Delta 1.2.0 and Spark 3.2.0, demo recording exists despite a live-screen hiccup.
AI-written summary · about
chapters (17)
- 0:00Multidimensional indexing intro
- 1:57What is an index and why it matters
- 3:43How multidimensional indexing works
- 7:20Indexing unstructured files
- 9:53Lakehouse formats and Cube's role
- 12:03Directory structure and metadata
- 16:59Operations with index and sampling
- 21:19Advanced Spark techniques
- 23:09Optimizing the multidimensional index
- 24:12Compacting small files
- 26:23QBeast roadmap
- 27:19Demo attempt and issues
- 31:14Question and answer session begins
- 31:35Performance with vs without QBeast
- 32:42QBeast production status
- 34:05QBeast future developments
- 36:54Closing remarks
related talks
46mOpen Table Formats in the Wild: From Parquet to Delta Lake and Back
05mAnnouncing Delta Lake 4.0 with Liquid Clustering
25mBuild open format data lakehouse
31mDelta Lake: Open Source Reliability and Quality for Data Lakes
25mDelta Lake has competition
28mLakehouse Architecture with Delta Lake and Databricks
34mCatalogs, Iceberg, and Delta: Building the Foundation for Lakehouse Analytics
40mDuckDB Meets Data Lakes