The Future of Metadata in Prometheus: Enhancing Storage and Usability
summary
Proposes a persistent metadata store that treats metadata as key/value pairs stored as separate indexed series rather than in-memory labels, with a Grafana Labs hackathon meta store prototype that successfully demos OpenTelemetry resource attributes ingested into Prometheus and queried via explicit meta matchers. Design options include storing metadata IDs on series, a lightweight series→metadata map, or metadata references in chunk headers; estimates a metadata change can briefly double series in memory but keeps churn limited by sharing. Uses a suggested __meta prefix for ingestion though interface remains undecided.
AI-written summary · about
chapters (15)
- 0:00Introduction and speakers
- 1:30What is metadata in Prometheus
- 7:57Designing a metadata store
- 12:52Ingestion formats and prototypes
- 16:04Demo of metadata store prototype
- 19:59How to get involved
- 21:04Q&A on metadata labels
- 22:49Metadata for calculations
- 24:14Exemplars and metadata
- 25:17Label naming conventions
- 26:24Stable metrics and deprecation
- 27:28Migration from infometrics
- 28:35Reserved metadata labels
- 29:38High cardinality risks
- 31:05Q&A wrap-up
related talks
17mHow Prometheus indexes Data and Why You Should Care
39mBuild a High Performance Remote Storage for Prometheus with Unlimited Time Series
46mPrometheus as a Time Series Database
55mAdding ++ to Prometheus
37mMetaOps: Metadata Operations For End-To-End Data & Machine Learning Platforms
06mLightning Talk
48mMetadata for the Cluster: A Key-Value Heroes Race
21mThe Future is Bright, the Future is Prometheus Remote Write