Demystifying Benchmarks: How to Use Them To Better Evaluate Databases
summary
Tracks TPC history and shows why published scores no longer suffice, then demonstrates practical tool use: YCSB and sysbench/CISbench to drive workloads, iperf to measure network, plus hacked drivers for reproducing customer bugs. Offers concrete checks: plot a performance curve (throughput vs latency), identify the real bottleneck (CPU, network, driver — e.g. needing 13 drivers not 8), and warns that TPC went from ~500 tpm to ~50M tpm while published results can be gamed. Use these methods to verify claims and size systems correctly.
AI-written summary · about
chapters (7)
- 0:00Intro
- 0:34Demystifying Benchmarks Order Entry
- 7:00The Good: - Established the rules to the game - For the first time competitive performance claims could be compared - Audited results - Standard workloads focused the industry and drove performance improvements
- 18:46Know your objective. - What are you trying to measure/test?
- 21:10Choose your approach. Option 1: Rely on published Results
- 23:55A data point is not a curve. (Common mistake.) Performance Curves
- 28:02Understand where there's a bottleneck (Common mistake.) - Where systems can bottleneck
related talks
51mIntroduction to Fair-Use TPC Benchmarking Kits
31mDemanding the impossible rigorous database benchm
45mTuning Hardware and the Operating System: Benchmarking on PostgreSQL
29mPerformance Archaeology - 20 years of improvements
47mThe Road Towards an Open Database Performance Ranking
50mPerformance Archaeology
20mAutomate Database Benchmarking with Xbench
33mCassandra Performance Tuning Like You've Been Doing It for Ten Years