How We Test Performance in Rider
summary
Rider performance testing collects raw timing from TeamCity CI and a custom retrospective performance service that applies statistical tests (Mann–Whitney, Kolmogorov–Smirnov) to detect regressions — the system flagged 734 anomalies after an IDE update. Engineers run targeted experiments on a 12‑node Mac mini farm (4 macOS, 4 Windows, 4 Linux) to reproduce clustering, multimodal timing distributions, thermal throttling and cold versus hot starts, and they moved from fixed thresholds to adaptive, history‑based assertions and iterative sampling; fixes often reduce variance but large‑scale automation and flaky hardware remain challenges.
AI-written summary · about
chapters (9)
- 0:00Performance testing intro
- 2:33Performance issues
- 4:19Monitoring and analysis
- 13:12Building test system
- 18:11Test failure criteria
- 22:45Physical farm testing
- 29:52Real user simulation
- 31:40Team performance culture
- 34:12Q&A
related talks
49mPerformance Testing
44mPanel: Performance... performance
53mEarly Performance Testing: Finding News You Can Use
35mPlugin Testing: Performance, UI, and Functional Testing
29mProgressive Testing to Meet the Performance Imperative
40mWhy component level performance testing should always be part of CD and CI
42mAdding Performance Assertion to Standard Functional Testing
37mAutomated Performance Testing With WebDriver