Cluster Golden Signals to Avoid Alert Fatigue at Scale
summary
Intuit builds a cluster-level observability layer that reduces alert noise across 275+ clusters, 20k namespaces, 2,500 production services, aggregating Prometheus metrics into four “cluster golden signals” (errors, latency, traffic, saturation). Alerts use component-specific rules (success-rate SLAs, error-count thresholds) and an autoencoder-based anomaly pipeline via Prometheus + NumaFlow/NumaLogic to derive per-cluster baselines. A demo of an AWS VPC CNI failure shows cluster signals detect platform issues in 5 minutes vs 60 minutes for service-only alerts, and the system runs in production.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 7:15Defining golden signals
- 19:05Demo: golden signals in action
- 25:58Static thresholds vs anomalies
- 31:58Future: NumaPraj AI detection
- 36:36Key takeaways
- 37:22Q&A
related talks
33mAlerting in Cloud Native Environments
18mAlert Analytics
31mAlerting for Distributed Systems—A Tale of Symptoms and Causes, Signals and Noise
43mFinding the Golden Signals with Prometheus
29mParanoid Observability –Success Stories and Misfortunes observing an observability stack
26mAIOps: New Approaches to Reduce Alert Noise
36m"Alert overload: How to adopt a microservices architecture..."
22mHidden Signals in K8s Clusters