Stable and Accurate Health-Checking of Horizontally-Scaled Services
summary
Fastly builds a distributed health checker that classifies an instance unhealthy only when its behavior deviates significantly from the cluster, avoiding cascading failures when all nodes slow under load. Implementation uses a three-stage filter: moving average denoising (example window ~20) to ignore transients, mean+std anomaly thresholding with a tunable alpha to compare error rate and latency (observed jumps from ~0.25 to ~0.5 ms), and sharp hysteresis dual thresholds. Filters run per-host, inputs multicast/gossiped, samples archived in BigQuery; CPU cost is negligible (~1%) for tens of hosts.
AI-written summary · about