Bad Metrics and Very Bad Decisions
summary
Shows that common privacy and risk metrics routinely mislead: a company goal to “send N notifications” is trivially gamed, a time-to-first-response metric produced polite but useless replies until measurement stopped, and boosting a connections count increased harassment and broke access controls. Detection-rule quotas diverted effort from real threats. Practical fixes include using sampling guard metrics, explicitly stating action an indicator should trigger, rejecting uncomputable metrics, and tracking uncertainty and adversarial manipulation so numbers actually drive safer decisions.
AI-written summary · about
chapters (6)
- 0:00The dangers of bad metrics
- 1:25Defining perverse metrics
- 2:36Measuring effort vs results
- 5:11Gauges, levers, and proxies
- 7:33The impact of uncertainty
- 11:04Strategies for better metrics
related talks
26mMetrics and Modeling – Helping Teams See How to Improve
53mYour Attention, Please: Optimizing Metrics for Humans
29mYour Metrics Are Lying: How to Stop Reporting and Start Learning
14mWhat dashboards don't tell you
43mLooking to social science for help with metrics
12mGamify the Metrics, Amplify the Impact
46mInside Payback: Using Metrics to Demonstrate Insider Risk Program Value
05mPerception and bias and metrics, oh my!