Avoiding the Ordering Trap in Systems Performance Measurement
summary
Performance results often depend on trial order, not just configuration: a literature survey of 56 reproducible papers finds only 7% specify reset procedures, and a 9‑month dataset of 2.3M measurements from ~1,700 servers shows ordering effects for the majority of experiments. The authors provide a repeatable method using random vs fixed runs, nonparametric testing (referred to in the transcript as the Crystal Wallace test with Bonferroni correction), report ~8% mean absolute differences and a case with ~17% shift, and release the OrderSage tool to detect and mitigate these biases.
AI-written summary · about
chapters (11)
- 0:00Ordering trap in performance
- 1:02Memory allocation variability
- 2:29Ordering effects bias conclusions
- 3:32Root causes of ordering trap
- 4:14Strategies to avoid ordering trap
- 5:41Literature on ordering effects
- 7:50Experiment design methodology
- 11:25Analyzing long-term data
- 13:52Introducing OrderStage tool
- 14:34Case studies on ordering impact
- 15:59Summary and tool release with Q&A
related talks
41mAdvanced Napkin Math: Estimating System Performance from First Principles
51mDesigning for Performance
15mPrinciples and Methodologies for Serial Performance Optimization
01h 00mMeasure, Then Build
44mBehavioral Software Engineering
50mTest Intelligence for Architects
42mAdventures in Efficiency and Performance
41mOptimizing for Value - Fight Failure Demand With Bottlenecks