Fallacies of Agreement: A Critical Review of Consensus Assessment Methods for Gesture Elicitation
summary
Claims that common gesture-agreement measures (A and AR) reliably show consensus are false: reanalysis of four studies finds large discrepancies and shows raw agreement ignores chance, with Fleiss’ kappa and Krippendorff’s alpha returning values near zero and confidence intervals that include zero for some datasets (paper-phone). Statistical tests Vrd/Vb produce very high Type I errors (over 40% at α=0.05). Recommends reporting percent plus chance-corrected indices, visualizing bias distributions, and using jackknife/bootstrap interval estimates instead of fixed thresholds.
AI-written summary · about
chapters (24)
- 0:00Introduction to paper
- 0:26What is gesture elicitation
- 1:30Understanding agreement in studies
- 2:16Agreement indices A and AR
- 3:20Problems with current agreement metrics
- 4:02Case study on paper phone study
- 5:27Kappa and alpha for agreement
- 6:51Confidence intervals and uncertainty
- 7:35Re-analysis of four CHI studies
- 8:18Why kappa statistics differ
- 9:40Sign vocabularies and bias
- 11:28Mathematical definition of bias
- 12:34Bias distribution examples
- 12:56Recommendations on reporting agreement
- 13:18Interpreting agreement thresholds
- 14:45Avoiding prescribed thresholds
- 15:06Significance tests Vrd and Vb
- 16:12High Type 1 errors in tests
- 16:58Permutation test results
- 17:39Resampling techniques for estimates
- 18:21Importance of data sharing
- 19:05Acknowledgments and conclusion
- 20:11Q&A on impact and scale
- 21:38Q&A on qualitative vs quantitative