Performance tests and other lies
summary
Eight practical anti‑patterns and concrete fixes: use percentiles and histograms instead of averages, prefer JMH for Java microbenchmarks and combine JMH with profilers, and don’t trust a single profiler — sample vs instrumenting differences matter so run multiple tools like pyroscope + async-profiler. Design tests by goal (latency vs throughput vs stress vs endurance), avoid coordinated omission with open‑model load generators or queuing schedulers, watch result variance (a “40% faster” jump once turned out to be 404 errors), and always correlate with observability.
AI-written summary · about
chapters (18)
- 0:00Introduction and welcome
- 4:24Why performance matters now
- 5:51Eight anti-patterns overview
- 6:14Numbers game in performance tests
- 8:43Bench marketing explained
- 10:11Statistics in performance testing
- 11:59Profilers and their lies
- 17:24Asking the wrong questions
- 21:41Latency vs throughput
- 24:32Modeling workloads correctly
- 29:06Database problems everywhere
- 31:20Result variance and noise
- 33:47Testing environment design
- 34:31Load generator pitfalls
- 41:53Measuring time takes time
- 44:02Looking in the wrong direction
- 46:12The final boss: no tests
- 47:56Q&A session
related talks
02h 44mMeasure, don’t guess: benchmarking stories from the trenches
02h 41mMicro Benchmarking - The Art of Realizing One is Wrong
08m“8 Performance testing tips that every Test Engineer should know.”
08mCommon Mistakes in Performance testing
49mPerformance Testing
50mPerformance Testing Java Applications
06mIGNITE: How to benchmark … poorly
40mWrite awesome tests