History-based Latency Prober Tuning
summary
Uses historical prober data to auto-tune alert thresholds by fitting log-normal fits or a skew-adjusted boxplot (metacouple skewness) and backtesting against known outages. Applies median windowing (e.g. 8-sample median) to cut noise, recalculates per-environment thresholds across hundreds of environments, and replaces the single hard threshold that had become an insane 12 second cutoff in one case. Claims reduced false positives, preserved sensitivity where it matters, and a monthly job can recompute and push new configs.
AI-written summary · about
chapters (12)
- 0:00Intro
- 0:29What are Probers
- 1:09Key Design Decisions
- 2:38How is it done in practice?
- 4:26A Better Solution: Look to History!
- 7:03Non-parametric approach
- 8:38Windowing (Average)
- 8:54Windowing (Median)
- 9:21An Alert?
- 9:39Another Example
- 10:01An Alert!
- 10:52Conclusions
related talks
36mHow to Include Latency in SLO-based Alerting
17mStopping Performance Regression via Changepoint Detection
39mBeyond Goldilocks Reliability
22mLancet: A self-correcting Latency Measuring Tool
15mExploring the math of thresholds
38mWatch Everything, Watch Anything: Anomaly Detection
19mSensory Friendly Monitoring: Keeping the Noise Down
50mApplied Alerting Philosophy