Evaluation and Refinement of Clustered Search Results with the Crowd
summary
A crowd-powered workflow called Refinery workflow evaluates and refines clustered search results for 120 Google Play queries by generating clusters from four algorithms, having 50 MTurk workers rate each clustered result (about 40 ratings needed after filtering), then using 20 workers to suggest merges or deletes and 20 to pick cluster titles. Expert comparison on 57 production queries shows crowd choices match experts and after-refinement results rate better 83% of the time, cost less than expert work but not by an order of magnitude, and scales to thousands but not millions.
AI-written summary · about
chapters (8)
- 0:00Start
- 5:10The Refinery
- 5:57Evaluation
- 7:54Summary
- 9:57Interface for Merging Two Clusters
- 11:08Crowd Evaluation
- 15:39Summary of Our Findings
- 16:17Relative Costs
related talks
26mStreamlining Property Search Across Multiple Platforms
01h 48mRecommendation engines explained
41mDynamic Ranking of Search and Recommendation Results in High-Traffic Systems
14mTagRefinery: A Visual Tool for Tag Wrangling
09mOrganizing Chaos: The Growth of Collaborative Filters
44mSituationalization: Using situational data to optimize online shops
38mData Science from the Trenches
08mComparing Apples and Oranges: Human and Computer Clustered Affinity Diagrams Under the Microscope