Beyond TSDB: Unlocking Prometheus with Parquet for Modern Sc...
summary
Projects Mimir, Cortex and Thanos replace Prometheus TSDB with Parquet columnar storage, storing one series per row and sharding chunks into 8‑hour chunk columns. Benchmarks show dramatic savings: a sample block of 13M series shrinks from 26GB chunks + 4.6GB index to 14GB chunks.park and 153MB labels.park, queries drop from ~17s to ~3.5s, and the library reports ~90% fewer range calls, ~70% less memory and ~40% fewer allocations. A converter migrates existing TSDB blocks and promql acceptance tests pass.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 5:54TSDB index problems
- 12:01Parquet design and schema
- 13:26Parquet data lookup
- 20:35Storage efficiency vs TSDB
- 21:38Live demo results
- 22:41Q&A: New features
- 24:11Q&A: Query limits
- 25:15Q&A: Migration path
- 27:01Q&A: Small dataset benefits
- 27:42Q&A: Parquet structure
- 28:25Q&A: Future steps
related talks
37mPrometheus Deep Dive
23mHow we converted our Cortex data to TSDB blocks
30mTSDB: The Past, Present and the Future - Goutham Veeramachaneni, IIT Hyderabad
20mIntro to Scaling Prometheus with Cortex
01h 19mIntro + Deep Dive: Prometheus
17mHow Prometheus indexes Data and Why You Should Care
23mPrometheus – What it is, What is New, and What is Coming
39mHow To Take Prometheus Planet Scale: Massively Large Scale Metrics Installations