Building state-of-art LSM-tree compaction in ScyllaDB
summary
ScyllaDB implements incremental compaction by breaking SSTables into 160 MB fragments, letting compaction release input fragments as output fragments finish so tiered layouts no longer require ~100% extra free space. A hybrid tier/level layout compacts small recent runs like tiered storage while keeping the largest level leveled, trading write amplification against space amplification ~1.1 like leveled stores. A closed-loop controller measures compaction backlog per-strategy and adjusts compaction throughput, keeping p99 latency bounded while preventing compaction backlog and disk fill.
AI-written summary · about
chapters (18)
- 0:00Introduction
- 1:48ScyllaDB and LSM compaction
- 3:34LSM compaction importance
- 11:49Compaction strategies trade-offs
- 14:00Run conjecture in compaction
- 15:23Cassandra vs Scylla compaction history
- 16:30Tiered compaction deep dive
- 21:29Leveled compaction deep dive
- 33:11Tiered vs leveled compaction challenges
- 37:53Hybrid and incremental compaction
- 43:34ScyllaDB incremental compaction
- 47:05Hybrid compaction benefits
- 51:18Tuning hybrid compaction
- 52:42Time-window compaction overview
- 53:44Self-tuning compaction theory
- 57:41Compaction controller in practice
- 1:00:11Controller performance adaptation
- 1:01:16Summary
related talks
29mThe Unified Compaction Strategy in Cassandra 5
26mDesigning a Query Queue for ScyllaDB
41mPerformance & Fault Tolerance: Building a Modern Database
30mOvercoming Distributed Databases Scaling Challenges with Tablets
24mTSDB Data Compactions on Scale Without Headache: Now & Future
35mHow to Save on Scaling by Moving from Cassandra to Scylla DB
18mPreview: KV-Stores and Databases
39mCompaction for All! Unified Compaction Strategy in Cassandra 5.0 (CEP-26)