Lightning Talk: Instant Vectors ≠ Scalars - How it bit me
summary
Graphing a blackbox exporter across locations with Victoria Metrics triggers a surprising behavior: a binary-operator right hand side label push down applies host labels to a metric that has none, so count(exporter_location) becomes mismatched and returns no samples at certain granularities. The engine skips work when there are less than 100 unique values, so zooming changes results. Prometheus does not do this; the problem stems from Victoria Metrics converting an instant vector treated as a scalar, an issue that was filed and later closed.
AI-written summary · about
related talks
19mCutting Through Metrics Cardinality Noise with VictoriaMetrics
36mVictoriaMetrics: scaling to 100 million metrics per second (EN)
40mScale Your Monitoring Solution With the VictoriaMetrics Ecosystem
11mLightning Talk: “Metrics That Lie”: Understanding OpenTelemetry’s Cardinality Capping...
38mAggregating Metrics In-Flight: Challenges and Opportunities
11mLightning Talk: Beyond Billions: Operating Thanos, Prometheus & OpenTelemetry...
41mSo Many Metrics, So Little Time: 5 Prometheus Exporter Best Practices
13mLightning Talk: A Drop-in System To Accelerate Metrics Observability by 100x Us...