Insights into Your Cloud Database: How Storage Engines Actually Work
summary
Explains how Hummock (RisingWave’s engine) actually writes and reads: MemTables flushed to SSTs, L0 sublevels and flush every second epochs create rolling versions, and compaction is a copy-not-move process that trades read vs write amplification. Gives concrete numbers and facts: 0.1% Bloom filters, about seven LSM levels, terabyte-scale clusters and hundreds of nodes, S3 shared‑disk storage, and stateless compactors runnable as separate workers or Lambdas. Useful if wanting implementation details, performance tradeoffs, and what was built and why.
AI-written summary · about
chapters (21)
- 0:00Storage engine internals
- 1:13RisingWave Labs architecture
- 3:45Slotted Pages in PostgreSQL
- 8:28Slotted Pages limitations
- 8:49LSM storage engines
- 10:15Reading LSM data with bloom filters
- 12:09Range queries and SSTables
- 13:57Time travel queries
- 16:05LSM compaction trade-offs
- 17:08Level-based LSM organization
- 18:36Tiered and level compaction
- 20:06Optimizing L0 reads
- 21:29Sub-levels and flush keys
- 23:23Compaction groups for workloads
- 25:32SSTable block storage
- 26:15Non-destructive compaction
- 28:02Epochs and snapshotting
- 30:57LSM metadata tracking
- 32:24Compaction tasks and roles
- 33:29Deploying compactors
- 34:55Q&A
related talks
20mBuilding An Elastic Query Engine on Disaggregated Storage
35mQuestioning Database Claims: Design Patterns of Storage Engine
34mWhat We Talk About When We Talk About Disk IO
39mDissecting the Humble LSM Tree and SSTable
18mPreview: KV-Stores and Databases
24mPerformance tuning in BlueStore & RocksDB
23mBuilding & Running a Scuba-Inspired Observability Tool
01h 08mApache Hudi: A Database Layer Over Cloud Storage for Fast Mutations & Queries