Using Change Point Detection to Fight Noisy Benchmark Results
summary
Change point detection locates statistically significant shifts in noisy benchmark time series using libraries like Hunter, MongoDB single processing library, and ruptures. Example: Hunter flags a clear regression in get parent PID latency after a Spectre mitigation was enabled for KVM guests. Algorithms require multiple samples (roughly five) before surfacing a change and can delay detection by days; comparison work is pairwise and can be computationally heavy (up to O(n^3)), but it successfully isolates real regressions from random noise.
AI-written summary · about
chapters (9)
- 0:00Introduction to talk
- 1:07Understanding benchmark noise
- 3:55Sources of noise in systems
- 6:07Limitations of visual analysis
- 8:15Threshold-based methods
- 9:18Change point detection overview
- 12:27Real-world example with Linux kernel
- 13:52Drawbacks of change point detection
- 15:42Further reading and conclusion
related talks
16mTaming Noisy Benchmark Results Using Change Point Detection
17mStopping Performance Regression via Changepoint Detection
12mHistory-based Latency Prober Tuning
49mNoisy Neighbor Detection: A New OSS Collector
35mCase Study: A Thundering Herd in the Wild
12mModel Monitoring: Detecting and Analyzing Data Issues
39mIt's (not) down in the noise.
56mWhy Your Go Benchmarks Are Lying