Fast on paper, slow in reality: what we got wrong about performance
summary
Built a strongly-consistent distributed key‑value store in Go on Badger LSM with hash partitioning and persistent TCP connections, then discover real bottlenecks: a single mutex-protected waiting list causes lock contention (fix: sharded waiting lists and cache‑line aligned lock‑free buckets), Two‑Phase Commit creates excessive socket writes so a single-leader fast path collapses begin/add/commit into one IO, scatter‑gather range queries cause OOM so pagination + client-side merge prevents merge pressure, outbound connector saturation is solved by parallel serializers and multiple TCPs — net result 115ms → 10ms latency and 10,000 TPS at 32 partitions RF=3.
AI-written summary · about
chapters (8)
- 0:00Introduction and motivation
- 1:21Architecture overview
- 8:06Benchmarking methodology
- 10:18Optimizing the waiting list
- 15:21Refining two-phase commit
- 20:03Range query handling
- 26:22Parallelizing serialization
- 30:25Conclusion and Q&A
related talks
19mBuilding a Performant Configuration-Store Layer
20mWriting High Performance Databases in Go
48mLightning Talks
35mQuestioning Database Claims: Design Patterns of Storage Engine
01h 03mExtreme Performance Architecture
46mEdge Performance with In memory NoSQL
37mMoving on from RocksDB to something FASTER
45mBeyond Lists: High Performance Data Structures