News Ninja: Gamified Annotation of Linguistic Bias in Online News
summary
News Ninja is a gamified annotation tool with a tutorial and five modes including Publish game mode and Quick Words that collects labels from players recruited on Prolific (100 participants). Study re-annotates 370 BABE sentences and 150 new sentences (five annotations each) in ~21 minutes per player, yielding Krippendorff’s alpha .44 on re-annotated BABE (+10.3% vs experts) and .39 on new sentences (90–109.5% gains vs crowdsourced baselines). The system uses instant and delayed feedback and improves some expert mislabels, but struggles with subtle bias and requires better incentives.
AI-written summary · about
related talks
29mAnnotating data the right way
33mAdvanced Natural Language Processing with Free Low-Code and...
37mOptimal on Paper, Broken in Reality
40mBuilding new NLP solutions with spaCy and Prodigy
17mNatural Language Processing in Automation
35mThe Podcast Pipeline Nobody Sees: Behind the Scenes with AI Agents and AI gen
43mGetting started with AI for free
18mReality is not an end-to-end prediction problem