Crowdsourcing Similarity Judgments for Agreement Analysis in End-User Elicitation Studies

on YouTube ↗15m

" summary

A crowd-powered web tool called Crowdsensus crowdsources pairwise similarity votes and runs five unsupervised clustering algorithms to group symbols from elicitation studies. It analyzes a dataset of 430 voice commands (10 referents × 43 commands), collects judgments from 410 crowd workers, and compares to six experts; the crowd matches expert consensus for seven of eight convergent functions, resolves one formerly disputed function, costs about $6 vs $5 per referent, and is four times faster (0.78 vs ≈3+ minutes).

AI-written summary · about

" chapters (9)

1. 0:00Introduction and background
2. 1:11What is an elicitation study
3. 2:38Research questions and Crowdsensus
4. 3:44Interface designs and pilot testing
5. 5:11How Crowdsensus works
6. 7:48Comparing crowd to expert results
7. 9:55Cost and time benefits
8. 10:40Tool access and acknowledgments
9. 11:24Q&A session