The Measure and Mismeasure of Fairness
summary
Argues that common algorithmic risk scores (logistic regressions, threshold rules, confusion-matrix metrics) can mislead: group-based formal measures like calibration and false positive rate fail to detect or can enable discrimination. Uses ProPublica/Broward estimates (roughly 30% vs 15% FPR gap) and a St. George’s admissions example to show an algorithm codifies bias, shows infra-marginality from differing risk distributions, and warns that satisfying formal metrics often produces perverse, discriminatory decisions.
AI-written summary · about
chapters (20)
- 0:00Quantifying bias in machine decisions
- 0:43Summary
- 1:51Pretrial detention A detailed case study
- 3:49Key assumptions
- 8:38From features to decisions
- 10:39Risk distributions
- 12:42From risk to decisions Threshold rules
- 19:46A double standard
- 20:30Fairness of a single threshold
- 22:07Popular mathematical definitions of fairness
- 23:31Discrimination with calibrated scores
- 28:27Classification parity
- 28:50False positive rate parity
- 29:11Error rate disparities in Broward County
- 32:18Calculating false positive rates
- 32:51Infra-marginality
- 33:56The problem with false positive rates
- 37:00Are the data biased?
- 37:32Biased labels
- 39:33Biased predictors
related talks
50mThe Measure and Mismeasure of Fairness
28mMeasuring Model Fairness
38mTracking and Improving Information in the Service of Fairness
40mQuestions and Practices to Make Algorithmic Decision Making More Fair
41mMeasures and Mismeasures of Algorithmic Fairness
42mFairness in Machine Learning: From Theory to Practice
31mMeasuring Model Fairness
53mGroups and Individuals in Fairness and Probability