Sparse High-resolution Histograms in the Prometheus TSDB
summary
Prometheus TSDB prototype stores sparse high-resolution histograms as single series per histogram with metadata encoding a growth factor 1.1 and a sparse run-length list of used buckets, samples encoded with delta/delta and XOR for sums. Benchmarks show ~index size ~17x reduction (93–94%), chunk size down 43–48%, overall storage 48–60% less. Observed bucket counts vary 1–128 observed; implementation scrapes, ingests and retrieves raw histograms, is experimental, and falls back to lowering precision when a configured bucket limit (default 150) is exceeded.
AI-written summary · about
chapters (18)
- 0:00Introduction
- 0:49Recap
- 2:00Improvements
- 3:53Design Dock
- 4:12Hackathon
- 4:25Heatmap
- 6:03Overview
- 6:30Sparse Buckets
- 8:22Histograms Chunk
- 11:05Test Setup
- 11:47Data Structure
- 13:17Data Snapshot
- 14:15Benchmark Results
- 15:53Conclusion
- 17:34Partitioning
- 18:21Limitations
- 19:32Future work
- 20:46Outro
related talks
23mImplementing Out-of-Order Native Histograms in Prometheus
24mDownsampling in Prometheus
17mHow Prometheus indexes Data and Why You Should Care
30mTSDB: The Past, Present and the Future - Goutham Veeramachaneni, IIT Hyderabad
25mQuantiles at Scale: Choosing the Right Estimation Algorithms for Observability
30mLifecycle of a Sample in the Prometheus TSDB
39mBuild a High Performance Remote Storage for Prometheus with Unlimited Time Series
55mAdding ++ to Prometheus