Lightning Talk: Can you Trust that Benchmark?
summary
Argues that benchmarks must be validated not trusted out of the box, showing a local Cassandra example where following practices—read errors, tune the benchmark, make runs reproducible and change only one parameter—raises CPU user time and raises throughput to 31,282 throughput. Recommends checking which resource is stressed (memory vs CPU), defining realistic targets (is 5% or 10x feasible), and treating run-to-run variance as a signal to investigate rather than ignore.
AI-written summary · about
chapters (6)
- 0:00Introduction
- 0:19What is benchmarking
- 0:40Misconceptions
- 1:27How to build trust
- 3:30Benchmark without practice
- 4:26Conclusion
related talks
06mIGNITE: How to benchmark … poorly
02h 44mMeasure, don’t guess: benchmarking stories from the trenches
35mBenchmarking: You're Doing It Wrong
38mYour benchmark may not guide a real application performance
02h 41mMicro Benchmarking - The Art of Realizing One is Wrong
46mStop Guessing and Start Measuring - Benchmarking in Practice
40mBenchmarking: Repeatable & Comparable
43mStop Guessing and Start Measuring - Benchmarking in Practice