Diabolic Database Design
summary
Builds an Erlang metrics database that trades durability for throughput, using per-socket batching, a cached Riak ring, mutable byte-array buffers (mk6), and sequential file writes so disks read/write contiguously. Claims a single node sustains 15–20M/sec and a five-node cluster hits ~9M/sec per node in tests, runs on ZFS or spinning disks, and accepts small data loss windows (seconds) as a design choice — the system exists and the benchmarks show it works.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:23The problem
- 6:24Metrics
- 9:00Specs
- 11:15Real Core
- 12:05Process
- 13:11Ring
- 14:29Dictionary
- 15:45Cash
- 22:19Storage Data
- 23:03Sequential Data
- 25:07Benchmark
related talks
29mThe Design and Evolution of Event-Driven Databases
28mAn introduction to Riak
16mScaleDB: A Scalable, Asynchronous In-Memory Database
30mCatching Waves with Time-Series Data
43mGoing Barking Mad with DalmatinerDB
40mOpen Riak how we gained from leaning on the BEAM
39mUsing Erlang, Riak and the ORSWOT CRDT at bet365 (...)
41mDesign (Failures) Case Study