Measuring Model Fairness
summary
Case study analyzes anonymized Civis Analytics client data (~130k rows, ~10k labeled African American, ~120k white) using a logistic regression classifier and four fairness metrics: disparate impact = 1 is fair, equal opportunity, equal miss opportunity, and difference in average residuals. Experiments inject measured label bias and sample bias (and both) and show metrics detect bias but also flag imbalance as bias, residual differences spot some biases better, and no single metric reliably isolates cause.
AI-written summary · about
chapters (12)
- 0:00Model fairness research
- 2:37Why fairness matters
- 5:29Biased recidivism case study
- 11:48Sources of bias
- 12:10Fairness metrics experiments
- 17:54Interpreting metrics
- 19:42Ethical responsibility
- 23:44Transparency laws demands
- 25:52Sampling imbalanced data
- 27:07Fairness without protected class
- 28:39Policy trade-offs
- 31:10Q&A
related talks
28mMeasuring Model Fairness
50mThe Measure and Mismeasure of Fairness
46mThe Measure and Mismeasure of Fairness
23mRemoving Unfair Bias in the Machine Learning Pipeline
46mA look at the methods to detect and try to remove bias in ML models
41mFairness in health AI: ensuring equity in healthcare cost.
01h 30mFairness Tales: How To Measure And Mitigate Unfair Bias in Machine Learning Models
36mMachine Learning Fairness: Lessons Learned