Questioning Database Claims: Design Patterns of Storage Engine
summary
Examines concrete storage choices: append-only WAL + fsync after every write vs batched fsyncs, B+tree indexes and Bloom filters for reads, and LSM with memtable flushes for write-heavy workloads. Recounts building a strongly consistent distributed key‑value layer, handling ~21,000 TPS (≈120,000 RPS) and moving from Postgres→Cassandra→in‑memory→kdb when Cassandra became costly at scale. Recommends Rocks/Pebble with fsync for audit logs, Bolt for single‑node config, and LSM engines for high write throughput.
AI-written summary · about
chapters (7)
- 0:00Introduction and context
- 3:13Storage engine foundations
- 5:54Pattern 1: Persistence
- 10:10Pattern 2: Efficient retrieval
- 14:43Pattern 3: Efficient ingestion
- 18:02Applying patterns to use cases
- 21:49Q&A and system architecture
related talks
42mContemporary Approaches to Data at Scale
52mHow shit works: Databases
34mWhat We Talk About When We Talk About Disk IO
39mAn Introduction to B-trees and LSM-trees for DEVs and DevOps
52mData storage for modern applications - Mike Lehan
29mThe Design and Evolution of Event-Driven Databases
18mPreview: KV-Stores and Databases
45mThe Database: A Sophisticated and Special Data Structure