Efficient and Consistent Time to Live TTL Implementation in MyRocks at Meta
summary
MyRocks adds native TTL with read filtering for immediate invisibility and a compaction filter for physical reclaim, prepending an 8‑byte TTL timestamp to every index. Benchmarks show TTL costs ~15 ms SQL CPU, 0 MB binlog and 300 ms wall time versus 20 s, 136 MB and 25 s for batched deletes. Replicas stay consistent by writing minimal snapshot metadata and using hybrid logical clocks; deployed fixes include RocksDB scheduler tweaks and guidance to avoid many secondary indexes or frequent TTL updates.
AI-written summary · about
chapters (13)
- 0:00TTL in MyRocks at Meta
- 1:27Ad-hoc delete jobs challenges
- 3:32Native TTL requirements
- 5:00Logical expiration via read filtering
- 7:50Physical expiration with compaction filters
- 10:42Snapshot isolation in TTL
- 14:13Replication challenges
- 18:09Hybrid logical clocks for TTL
- 19:31TTL overhead evaluation
- 21:42Adversarial workloads
- 23:30Production fixes for TTL
- 25:58Shard migrations with TTL
- 27:24Conclusion
related talks
10mGraduated Project Lightning Talk: Time to Live (TTL) Support for TiKV
52mRocksDB Performance and Reliability Practics
27mMyRocks: best practice at Alibaba - MySQL Storage Engine
38mExploring RocksDB in RGW: How We Manage Tombstones
24mPerformance tuning in BlueStore & RocksDB
28mComparing MongoRocks, WiredTiger and mmapv1 for performance and efficiency
15mRemoving Double-Logging with Passive Data Persistence in LSM-tree based Relational...
30mBuilding domain specific databases with a distributed commit log