Goodhart's law - pitfalls of measuring everything
summary
Explains Goodhart’s law as a practical hazard for TestCon attendees with four concrete failure modes: regressive correlation, extremal breakdown, adversarial perverse incentives, and causal confusion. Uses the Cobra effect and tester examples (ballooning test counts, automation effort versus capacity) and prescribes fixes: pairing indicators, authentic metrics like percent of requirements covered, premortems, keeping some measurements internal, and training teams so metrics measure progress instead of becoming the target.
AI-written summary · about
chapters (13)
- 0:00Goodhart's law intro
- 1:08Metrics need and pitfalls
- 5:05Goodhart's law defined
- 6:11Cobra effect perverse incentives
- 8:42Efficiency metric failure
- 10:32Four Goodhart flavors
- 11:34Regressive Goodhart automation
- 14:48Extremal Goodhart regression
- 16:55Adversarial Goodhart incentives
- 20:31Causal Goodhart correlation
- 21:36Counteracting Goodhart's law
- 24:47Long-term metric strategies
- 26:54Summary
related talks
23mYou Get What You Measure—Why Metrics Are Important
34mStop Running With Sharp Metrics
25mWhen Metrics Lie
42mThe Fault in Our Metrics: Rethinking How We Measure Detection & Response
37mBetter Reliability Through Observability and Experimentation
34mWhy Good Software Goes Bad
14mWhat dashboards don't tell you
01h 01mFinding Metrics that matter and using them safely