IGNITE: How to benchmark … poorly
summary
Vendor benchmarks pick favorable scenarios and claim universal wins, for example benchmarking a squid vs house cat use case or running Postgres vs MongoDB tests on experimental Lua hooks that nobody ships; running Kafka tests against an old Zookeeper‑dependent release to make the competitor look unfinished; exploiting defaults like MongoDB’s historical fire-and-forget socket writes or preloading optimized datasets. Advice: run continuous internal benchmarks to catch the slow‑boil performance regression and tune real production knobs rather than publish tuned marketing results.
AI-written summary · about
chapters (6)
- 0:00Start
- 0:30Benchmarketing
- 0:46Pattern: Find the "right" use-case
- 2:38Pattern: Tuned to your knowledge
- 3:10Pattern: "Optimized" defaults
- 3:42Pattern: Cheating
related talks
02h 44mMeasure, don’t guess: benchmarking stories from the trenches
05mLightning Talk: Can you Trust that Benchmark?
38mYour benchmark may not guide a real application performance
54mPerformance tests and other lies
46mStop Guessing and Start Measuring - Benchmarking in Practice
44mBenchmark testing
02h 41mMicro Benchmarking - The Art of Realizing One is Wrong
35mBenchmarking: You're Doing It Wrong