The Sixth Sense: Finding “Dead” Metrics and Silent Alerts in Your Monitoring
summary
Uses the VictoriaMetrics stack and its VictoriaMetrics Cardinality Explorer to show concrete savings: an empty Kubernetes demo generates 63,000 unique time series, with API server request_duration_seconds: 8,000 series (13%) that never registers a query and can be dropped to reclaim space. Explains programmatic detection of unused metrics, alerts that never fire by checking selected-series counts exposed as a no-match label, and contrasts CI validation with Cloudflare pint and real-time checks that catch deprecated metric names.
AI-written summary · about
chapters (16)
- 0:00Introduction to dead metrics
- 1:16Why observability is expensive
- 3:05Complexity of modern systems
- 5:31Observability vendors' approach
- 6:34AI's role in observability
- 7:14Default Kubernetes monitoring
- 8:37Cost of unused telemetry
- 9:40How metrics are used
- 11:04Finding unused metrics
- 13:56Demo of Cardinality Explorer
- 16:24Optimizing metrics usage
- 16:52Health of alerting rules
- 18:59VictorMetrics alerting page
- 21:26Complexity of monitoring
- 23:14VictorMetrics tools overview
- 23:57Q&A session
related talks
19mCutting Through Metrics Cardinality Noise with VictoriaMetrics
35mObservability’s Sixth Sense: Detecting Anomalies in Metrics
05mLightning Talk: How to detect and delete unused metrics?
40mObservability’s Sixth Sense: Detecting Anomalies in Metrics
28mHow to reduce expenses on monitoring with VictoriaMetrics
35mObservability’s Sixth Sense: Detecting Anomalies in Metrics
01h 00mHow do We Keep Metrics for a Long Time in VictoriaMetrics
11h 43mPercona Community Live Conference - Day 1 - Open Source Databases Conference