Measure, don’t guess: benchmarking stories from the trenches
summary
Shows concrete pitfalls and fixes: uses JMH harness plus perf and async-profiler to reproduce dead‑code elimination that reports absurd ops/sec, a black hole compiler-assist change that shifts timings by two orders of magnitude, and noisy‑neighbor runs that make error bars exceed 50%. Demonstrates monomorphic vs megamorphic dispatch costing roughly 2x, inlining limits (lowering inline depth causes ~10x slowdown), and CPU scaling traps; recommends many forks (≈10), warmups, and active benchmarking with counters and flame graphs.
AI-written summary · about
chapters (12)
- 0:00Introduction and presenters
- 1:56Why benchmarking matters
- 15:34Why not write your own benchmark
- 37:51Dead code elimination pitfall
- 57:14Black hole taxonomy
- 1:03:00CPU scaling effects
- 1:15:17Noisy neighbor problem
- 1:26:43Type profile pollution
- 1:56:06Inlining optimization impact
- 2:09:08Recursive inlining example
- 2:16:44Branch prediction and hash collisions
- 2:39:56Key takeaways and conclusions
related talks
43mStop Guessing and Start Measuring - Benchmarking in Practice
54mPerformance tests and other lies
46mStop Guessing and Start Measuring - Benchmarking in Practice
38mYour benchmark may not guide a real application performance
21mBenchmarking Like a Boss
39mMicro Benchmarks
49mHow Not to Do Benchmarking
06mIGNITE: How to benchmark … poorly