Why We Built Our Own Distributed Column Store
summary
A custom datastore called Retriever implements a disk‑based distributed column store inspired by Scuba, ingesting events from Kafka and serving sub‑second analytics at startup cost by using SSDs, per‑column files, and segment directories for quota/aging. It runs two replicas per Kafka partition, bootstraps with rsync and Kafka offsets plus checkpointed indexes, and uses HyperLogLog and t‑digest for count‑distinct and percentiles. It operates in production for paying customers.
AI-written summary · about
chapters (27)
- 0:00Intro
- 0:36Please meet Retriever
- 1:13Retriever is a special purpose data store
- 1:27What is Honeycomb?
- 1:58How Honeycomb works
- 6:26Honeycomb under the hood
- 7:30Our requirements
- 11:12Requirements - summary
- 12:41Retriever at a glance
- 13:18Retriever compared to Scuba
- 14:31Architecture - write path
- 15:06Architecture - read path
- 15:41Data model - datasets
- 16:11Data model - events
- 19:37Row oriented storage
- 21:22Column oriented storage
- 23:05Storage Format - timestamp column
- 23:43Storage Format - reading
- 27:53Distributed queries
- 28:56Distributed reads - calculations
- 31:38Distributed reads - fanout
- 33:01Detour - Kafka
- 34:56Ingestion
- 36:09Quota management
- 37:50Fault tolerance
- 38:32Failure recovery
- 40:34Bootstrapping new nodes
related talks
23mBuilding & Running a Scuba-Inspired Observability Tool
30mBuilding domain specific databases with a distributed commit log
42mColumnar Storage: Redefining Data Management for the Modern Era
14mA Cloud-native Architecture for Replicated Data Services
26mColumnar Storage: Redefining Data Management for the Modern Era
31mZero to data infrastructure
40mKalDB: A cloud native log search platform
32mDiscover the Modern Way to Build Real-Time Analytics Solutions at Any Scale