Writing a TSDB from Scratch: Performance Optimization
summary
VictoriaMetrics is a Go TSDB built from scratch and sustains about 17 million samples/sec ingestion with roughly 100 RPS reads, so optimizations focus on write-heavy, CPU-bound work. Techniques include string interning backed by sync.Map to cut allocations, a caching transformer for repeated label rewrites, limiting concurrency with token channels sized to CPU cores, and a bucketed sync.Pool of 17 buffer sizes to reduce GC and avoid oversized buffers. These measures reduce memory and CPU waste and work well for predictable, repeatable metric variants.
AI-written summary · about
chapters (13)
- 0:00Time series databases intro
- 1:04Time series data structure
- 3:11TSDB workload patterns
- 4:37TSDB system design
- 6:03String interning optimization
- 8:13Thread-safe string interning
- 10:43Caching function results
- 13:29Limiting CPU-bound concurrency
- 16:46Concurrency with token channels
- 18:14Byte buffer sync.Pool
- 21:02Bucketized sync.Pool
- 24:34Performance optimizations summary
- 25:18Q&A
related talks
35mWorking with filesystem in Time Series database
36mVictoriaMetrics: scaling to 100 million metrics per second (EN)
30mTSDB: The Past, Present and the Future - Goutham Veeramachaneni, IIT Hyderabad
31mHow ClickHouse® Inspired Us to Build a High Performance Time Series Database
24mTSDB Data Compactions on Scale Without Headache: Now & Future
40mScale Your Monitoring Solution With the VictoriaMetrics Ecosystem
44mOpenTSDB - Time Series Schema on Schemaless NoSQL
30mLifecycle of a Sample in the Prometheus TSDB