Moving on from RocksDB to something FASTER
summary
Compares RocksDB versus Microsoft Research’s FASTER as embedded state backends for streaming, benchmarking on EC2 SSDs with NEXMark queries and showing concrete tradeoffs: joins produce two orders of magnitude worse p99 latency on RocksDB, while a hand-tuned store is 1.6x faster than a generic FASTER wrapper; FASTER’s design uses a hybrid log + cache-optimized hash index to serve a hot in-memory set with in-place updates; a Rust wrapper and integration with Timely Dataflow exist and early results favor FASTER for point-heavy workloads but RocksDB can beat it for range-scan window implementations.
AI-written summary · about
chapters (9)
- 0:00Introduction to state store
- 1:00State in streaming systems
- 4:27RocksDB architecture
- 10:08FASTER architecture
- 17:44Implementing custom storage
- 22:28Benchmarking and performance
- 26:48Windowing implementation
- 31:03Lessons learned for Flink
- 35:32Q&A and closing
related talks
57mRocksDB
29mAccelerating Performance: Mastering RocksDB for High Speed Data Processing
20mEnabling Machine Learning through Real-Time Data Processing
24mSplinterDB: Closing the Bandwidth Gap for NVMe Key-Value Stores
52mRocksDB Performance and Reliability Practics
24mPerformance tuning in BlueStore & RocksDB
28mComparing MongoRocks, WiredTiger and mmapv1 for performance and efficiency
01h 03mExtreme Performance Architecture