Exploring RocksDB in RGW: How We Manage Tombstones
summary
RocksDB tombstones in RGW cause list operations to stall: tens of millions of tombstones accumulate, iteration latency rises up to ~6,000× when non‑compacted, and CPU hits 100% for hours. Collected the skip count metric from each OSD, pushed to Prometheus, visualized in Grafana, and built an automated job that triggers compaction (concurrent limit three) which drops CPU and HTTP 499/503 errors. SST dump tooling shows three prefixes hold >80% of tombstones, so a range-based compaction using a count-min sketch to estimate per‑prefix deletes is proposed.
AI-written summary · about
chapters (9)
- 0:00Introduction to Ceph Storage
- 1:20RocksDB in RGW
- 1:49Tombstone impact
- 4:35Quantifying tombstone issues
- 6:23RGW list internals
- 11:44Automatic compaction system
- 19:03Compaction trends
- 20:14Prefix-based optimization
- 27:13Q&A session
related talks
52mRocksDB Performance and Reliability Practics
57mRocksDB
51mSix New Important RocksDB Features And Planned Works
29mAccelerating Performance: Mastering RocksDB for High Speed Data Processing
29mEfficient and Consistent Time to Live TTL Implementation in MyRocks at Meta
24mPerformance tuning in BlueStore & RocksDB
39mHow we isolate streaming ingest from search using RocksDB
29mCharacterizing, Modeling, and Benchmarking RocksDB Key-Value Workloads at Facebook