The Measure and Mismeasure of Fairness
summary
Argues that common group-based fairness metrics are unreliable: calibration can hide discrimination by withholding distinctions, and equalized false positive rates force perverse thresholds. Uses pretrial risk tools (thresholding a risk score) and shows mechanically how different risk distributions produce disparities even with perfect prediction — ProPublica-style numbers: 30% vs 15% false positives for Black versus White defendants in Broward County. Points out biased labels (arrests ≠ crimes) and that including protected attributes can sometimes reduce harm.
AI-written summary · about
chapters (14)
- 0:00Quantifying bias in machine decisions
- 1:00Summary
- 3:32Algorithmic risk assessment An example: the Public Safety Assessment (PSA)
- 4:58Key assumptions
- 9:16From features to decisions
- 10:31Risk distributions
- 15:38From risk to decisions Threshold rules
- 18:30Fairness of a single threshold
- 19:48Popular mathematical definitions of fairness
- 29:22False positive rate parity
- 30:57Calculating false positive rates
- 36:10The problem with false positive rates
- 40:57Biased predictors
- 44:43Algorithms policy
related talks
46mThe Measure and Mismeasure of Fairness
28mMeasuring Model Fairness
42mFairness in Machine Learning: From Theory to Practice
31mMeasuring Model Fairness
38mTracking and Improving Information in the Service of Fairness
01h 10mJournalist-in-Residence Talk: The Limits of Fairness
53mGroups and Individuals in Fairness and Probability
41mMeasures and Mismeasures of Algorithmic Fairness