Big(ger) Sets: Making CRDT Sets Scale in Riak
summary
Riak ships CRDTs but the naive Riak Data Types set makes writes and reads cost grow with set size; engineering replaces one-key-per-set with per-element keys in Big Set stored in LevelDB, uses observe-remove semantics with compact version vectors and delta replication, and adds buffered streaming plus no-op key decoding. Result: add/remove latency stays flat as sets grow (benched against a 78 million‑element example), reads become streamable and cheap membership checks work — the redesign scales and performs.
AI-written summary · about
chapters (10)
- 0:00Start
- 6:32Time Stamp Based Reconciliation
- 6:56Semantic Reconciliation
- 7:21Convergence
- 11:39Two-Phase Set
- 12:46Unique Identifier Set
- 16:22Version Vectors
- 30:35Delta Replication
- 34:28Reading a Big Set
- 35:47Membership Query
