Six New Important RocksDB Features And Planned Works

on YouTube ↗51m

" summary

Six concrete additions: a user‑space persistent cache that appends compressed blocks to log files, evicts whole files and maps block IDs in memory; direct I/O options that bypass OS page cache for reads and/or compactions; SST bulk loading to ingest pre‑sorted files directly into low levels and avoid compaction CPU; an insert‑with‑hint prefix table (~250 bytes/prefix) that speeds M‑table inserts (benchmarks show ~75% faster); a range‑delete API that writes range tombstones; blob storage that cuts write amplification from ~10 to ~1 for large objects.

AI-written summary · about

" chapters (40)

1. 0:00Intro
2. 1:35Mission of RocksDB
3. 6:03Leverage Tiered Storage
4. 6:37Caching Strategy
5. 9:38Persistent Cache: Eviction
6. 10:04Persistent Cache: lookup
7. 10:38How to Use Persistent Cache
8. 11:28What is Direct I/O? • In Linux, it is to use O_DIRECT option
9. 12:18Precondition to consider Direct I/O
10. 13:14Benefit of Using Direct I/O
11. 14:44Use Direct I/O to by pass OS page cache
12. 15:48Future Plan
13. 17:17Use Case 1: Shard Migration
14. 17:53Import from Hadoop
15. 18:37Bulk Loading vs. Normal DB Writes
16. 19:51Bulkload - Create SST file
17. 21:16RocksDB Write Path - Memtable
18. 21:54RocksDB Write Path - Flush
19. 22:59Bulkload - Ingest SST file
20. 24:04Bulkload - memtable conflict
21. 25:35Time-series data
22. 27:06Reduce CPU usage for binary search
23. 28:00Insert With Hints: Implementation
24. 29:09Insert with hint: How to enable
25. 29:41Insert with hint: Overhead and Limitaton
26. 30:39Benchmarks
27. 32:36Range Deletion Challenges
28. 35:12DB::DeleteRange(): New API
29. 36:38Blob Storage: Motivation
30. 37:16Blob Storage: Blob Files
31. 38:58Blob Storage: Garbage Collection
32. 40:01Benchmark - Write Amplification
33. 40:36Bulk Loading: Work Status
34. 41:25Recap: Compaction A RocksDB operation that merges SST files
35. 41:40Recap: Compaction Filter A customizable logic to delete key-value pairs during compaction
36. 42:24Manage Compaction Filters
37. 43:27Lua Compaction Filter
38. 44:16Example lua::RocksluaCompactionFilter Options lua_opt
39. 45:12Recap: Six New Features
40. 46:02Future of RocksDB

" related talks

• 52mRocksDB Performance and Reliability PracticsYoshinori Matsunobu · Percona Live 2022
• 11mEvolution of Development Priorities in Key-value Stores Serving Large-scale ApplicationsSiying Dong, Andrew Kryczka, Michael Stumm, Yanqin Jin · FAST '21
• 57mRocksDBMark Callaghan, Igor Canadi · The Databaseology Lectures (Fall 2015)
• 29mAccelerating Performance: Mastering RocksDB for High Speed Data ProcessingVarunkumar Nagarajan · Great International Developer Summit 2024
• 24mPerformance tuning in BlueStore & RocksDBLi Xiaoyan · Cephalocon APAC 2018
• 19mA zone-aware cache system for distributed databasesKuankuan Guo · Linux Plumbers Conference 2022
• 46mBlueStore V2 - an Evolutionary Step ForwardMark Nelson, Adam Kupczyk · Cephalocon 2023
• 29mCharacterizing, Modeling, and Benchmarking RocksDB Key-Value Workloads at FacebookSiying Dong, Zhichao Cao, David H.C. Du, Sagar Vemuri · FAST '20