Misuse, Misreporting, Misinterpretation of Statistical Methods in Usable Privacy and...
summary
Systematic review of 121 SOUPS papers over 20 years audits use of NHST and finds concrete failures: 23% use incorrect tests that ignore sample dependencies, 51% leave tests unverifiable by omitting test names or key details, misuse appears in 65% of papers, misreporting in 95%, and misinterpretation in 45%. 44% of assertions are only partially supported, 47% are invalid, and 58% lack information to evaluate claims; authors offer a reporting checklist and concrete recommendations.
AI-written summary · about
chapters (7)
- 0:00Statistical misuse in research
- 1:54Statistical testing pitfalls
- 3:17Statistical tests misuse
- 5:45Systematic review findings
- 7:49Impact on research assertions
- 10:14Recommendations for authors
- 10:59Conclusion
related talks
42mIs Your LLM Evaluation Missing the Point
24mThe more we test, the worse?
54mPerformance tests and other lies
41mA Human-Centric Quality Assurance Process for Open Source Softwa...
17mPrivacy in Theory, Bugs in Practice: Grey-Box Testing for Differential Privacy Libraries
50mWhy Everything Is Spoiled
19mThe Case for Auditing Algorithms: Disloyalty and Bias M
01h 21mEmpirical methods fairer AI - Auditing a Dutch public sector risk profiling algorithm