Keyvi the Key Value Index Optimized for Size and Speed #bbuzz
summary
Keyvi is a finite-state key value index used by Cliqz to serve ~6,000 requests/min with ~60 ms average latency and 99% availability. It stores 1.8 billion URLs and ~13 billion key/value pairs in a ~2.5 TB index, down from a 5 TB Redis deployment, by compressing shared prefixes/suffixes and using sparse-array persistence, mmap shared memory and multi-process serving. Lookups run in O(length of key), construction uses external sort plus a bounded LRU-backed minimizer, supports segment-delta updates, approximate/completion matching and SSD-backed paging.
AI-written summary · about
chapters (13)
- 0:00Introduction to Kiwi
- 1:01Performance and scale metrics
- 2:00Transition from Redis to Kiwi
- 4:00Architecture and benefits
- 5:30Compression and benchmarks
- 7:34Deep dive into lookup logic
- 10:03Memory management comparison
- 12:38Disk vs. RAM and SSD performance
- 15:01Understanding the finite state
- 18:00Persistence and data structures
- 22:35Values and data storage types
- 24:17Updates and advanced usage
- 27:27Q&A and concluding remarks
related talks
13mDifferentiated Key-Value Storage Management for Balanced I/O Performance
18mSplitKV: Splitting IO Paths for Different Sized Key-Value Items with Advanced
06mKeynote: Cut Database Costs, Fund Innovation with Valkey
36mWhat’s New in Valkey
48mValkey clusters unboxed
33mDiscover Valkey: The Path to Now
49mvalkey-bundle: One stop shop for real-time applications
10mThe unreasonable effectiveness of key value databases