Don’t Trust Your Profiler: An Empirical Study on the Precision and Accuracy of Ja...
summary
Six Java samplers (async-profiler, Honest Profiler, Java Flight Recorder, JProfiler, perf, YourKit) run each benchmark for 300 iterations repeated 30 runs on the Are We Fast Yet suite to measure precision, agreement and overhead. Average overhead stays low at 1%–5.4%, but precision varies: most profilers reproduce the same hottest method across runs yet some benchmarks show dramatic disagreement — DeltaBlue yields 8 different top methods with method shares ranging ~5%–29% and the top‑5 set across profilers contains 28 distinct methods. Safe‑point bias reduction does not eliminate these inconsistencies.
AI-written summary · about
chapters (13)
- 0:00Profiler study introduction
- 1:51Research questions and challenges
- 3:58How sampling profilers work
- 6:42Prior profiler accuracy work
- 8:25Testing profiler self-consistency
- 10:10Experimental setup and benchmarks
- 11:57Worst-case profiler inconsistencies
- 13:23Average-case profiler inconsistencies
- 14:51Disagreement between profilers
- 15:58Profiler overhead measurements
- 17:23Compiler non-determinism effects
- 18:25Summary of findings
- 19:51Q&A
related talks
52mProfilers Are Lying Hobbitses
50mJPoint 2023 TechRadar: Performance Tools
02h 41mMicro Benchmarking - The Art of Realizing One is Wrong
41mUnleash the Power of Open-Source Java Profilers
54mPerformance tests and other lies
33mWriting a Java Profiler in Pure Java
38mYour benchmark may not guide a real application performance
45mHow difficult can it be to write efficient code?