Triage Score: A Counterfactual Risk Assessment Instrument
Talkat Bridging Prediction and Intervention Problems in Social Systems (Simons Institute, 2026) · 2026-01-12..01-18·
summary
Proposes a triage score that replaces a risk-only prediction with expected counterfactual utilities and shows point identification under an additive counterfactual utility assumption. Uses a randomized rollout of the PSA in Utah with about 7,000 arrest cases, AIPW estimators and neural nets, conditions on judge-facing affidavit text via open-source LLM embeddings (≈6,000 dims), and models judge decisions as single-blinded mediators. Finds PSA changes judges’ cash-bail choices (fewer releases follow AIs) and yields more ambiguous policy rankings than risk scores, motivating tailored utility elicitation.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 1:08From evaluation to implementation
- 3:07Selective labels problem
- 4:21Triage score framework
- 5:32Decision theoretic foundations
- 8:59Public Safety Assessment case study
- 14:39Evaluating algorithmic recommendations
- 17:23Risk versus triage score
- 22:32Statistical challenges in utilities
- 25:19Pre-trial risk assessment
- 29:51LLMs for unstructured data
- 31:01Policy evaluation and decisions
- 33:26Preliminary results
- 37:44Conclusion
related talks
38mTracking and Improving Information in the Service of Fairness
46mThe Measure and Mismeasure of Fairness
33mAssurance Scoring Using Machine Learning and Analytics to Reduce Risk in the Public Sector
30mMaking Privacy Assessments More Efficient
28mAI in healthcare - Deep learning for precision psychiatry
14mBuilding with the Responsible AI toolkit
13mCan AI design a fairer future?
40mAI-Powered! Or Is It Just Hype?!