Detecting the Dip: Building Observability Around...
summary
Built an anomaly-alert system on Prometheus metrics that predicts per-client traffic using a median of three days (due to three‑day retention) plus a trend correction, and computes adaptive lower bands from a smoothed per-client standard deviation with a fixed 40% margin fallback and historical anchors (1 and 3 days ago minus 2σ). Alerts use a dual gate (prediction + lower band), tiered burn‑rate thresholds and windows; result: caught a medium‑traffic regression ~90 minutes after deploy and a low‑volume bank degradation in 30 minutes.
AI-written summary · about
chapters (11)
- 0:00Introduction and background
- 1:08Pain points with integrations
- 3:57Defining observability goals
- 5:21Data collection initiative
- 7:50Anomaly detection basics
- 10:21Predicting traffic patterns
- 12:26Calculating alert bands
- 16:21Alert thresholds and burn rates
- 19:13Dual-gate alert logic
- 21:19Real-world incident detection
- 22:41Key takeaways and Q&A
