Taming Noisy Benchmark Results Using Change Point Detection
summary
Change Point Detection identifies stable shifts in noisy benchmark time series so you can separate real regressions from random spikes, requires roughly needs ~5 samples to detect a shift, and often shows a detection lag (a change seen on Friday may have occurred Tuesday). Algorithms do pairwise comparisons with heavy compute, typically O(n^2) cost, and scale to a few hundred points with millisecond latency but become impractical at thousands. Open-source options mentioned include BR in Python, Hunter, and ruptures.
AI-written summary · about
chapters (9)
- 0:00Noisy benchmarking challenges
- 2:45Benchmarks inconsistent results
- 5:12CPU frequency governor noise
- 6:57Change point detection
- 8:40CPD algorithm components
- 10:45CPD limitations and requirements
- 12:30Computational cost and tools
- 14:17Open-source CPD libraries
- 14:50Summary
related talks
16mUsing Change Point Detection to Fight Noisy Benchmark Results
37mTracking Regressions and Improving UI Performance using Benchmarking Approach
39mIt's (not) down in the noise.
01h 10mPrincipled Performance Analytics
47mUsing Benchmarks for Tracking Regressions and Improving UI performance
47mBenchmarking C++, From video games to algorithmic trading
38mKnow your data: The stats behind your alerts
41mKnow your data: The stats behind your alerts