Can we do better than git s Merkle DAG?
summary
Argues that replacing git’s Merkle-DAG with Merkle-ized databases fixes bloat and useless cryptographic sediment by hashing and compacting LSM SSTable chunks deterministically, so replicas produce bit-identical states and can state-first sync rather than replay long logs. Implements RDx-style merge operators inside Pebble DB (project chky), plus a low-level C library; one production deployment counts money and reports up to 100× compaction of counters. Key techniques: deterministic compaction, hashed SSTable chunks.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 2:11Merkle DAG architecture overview
- 4:18Problems with Merkle DAG design
- 6:47LSM databases overview
- 9:21CRDTs in LSM databases
- 10:50Deterministic compaction
- 12:39Hashing chunks and sync
- 14:30Merkleized vs blockchain
- 16:40LSM database benefits
- 17:43Merkleizing database internals
- 18:24Q&A: Existing projects
- 19:27Q&A: Provability and proofs
- 20:45Q&A: CRDTs and banking
- 23:34Q&A: Mergeable JSON tools
- 24:59Q&A: CRDT use cases
related talks
52mThe Computer Science behind a modern distributed data store
42mBeyond Blockchain: Convergent Consensus
31mSecure Distributed Storage
42mJepsen 9: A Fsyncing Feeling
39mEventually Consistent Data Structures
01h 13mCRDTs: The hard parts
01h 12mScaling distributed ledgers: Achieving arbitrary throughput and sub-second latency
20mFine Grained Replicated State Machines for a Cluster Storage System