Experiments in Backing Prometheus with Clickhouse
summary
Builds a Remote Write/Read bridge that inserts Prometheus samples into ClickHouse and replays a week of real queries against 24 hours (~850 GB) of production data on a 92‑core node. Naive schema peaks at ~1.8 TB then compacts to ~1 TB via merge tree parts; switching to nested fields yields ~380 GB (about half Prometheus’s ~750 GB) but increases query latency (often ~4× for heavy queries). Translating regexes to native OR/prefix/LIKE and adding an n‑gram BFV index (n=5) cuts tail latency while adding ~100 GB.
AI-written summary · about
chapters (12)
- 0:00Introduction to ClickHouse
- 1:06Overcoming Prometheus limits
- 2:19The Prometheus bridge
- 3:15Remote read and write
- 3:53Benchmarking setup
- 8:30MergeTree performance
- 12:05Low cardinality fields
- 14:18Nested field strategy
- 17:09Encoding analysis
- 20:00Optimizing query patterns
- 24:31Results and future work
- 25:56Q&A session
related talks
37mFinding Bottlenecks in ClickHouse: Profiling, Heatmaps & Flamegraphs
39mHow To Take Prometheus Planet Scale: Massively Large Scale Metrics Installations
36mPrometheus - Intro, Deep Dive, And Open Q+A
27mFast DIY O11y with OpenSrc Analytics & Visualization
17mBuilding Real-Time Data Lake with ClickHouse
33mIntro: Prometheus
28mIntegrating Long-Term Storage with Prometheus [A]
36mDoing Things Prometheus Can’t Do with Prometheus