Increasing the Speed and Accuracy of Data Labeling Through an AI Assisted Interface
summary
An AI-assisted labeling interface uses a semi-supervised model to Rank, Annotate and Reduce label choices and initially shows the top five predictions with a View more expand option. A 54-person study on 100 short texts and 21 labels finds baseline accuracy ~72%, weak-assist ~79% and strong-assist ~78%. Assistance shortens time when the correct label appears in the top five but greatly increases time when users hit View more. The weak model is overconfident and prompts more scrutiny; calibrated confidence and consistent spatial ordering improve outcomes.
AI-written summary · about
chapters (3)
related talks
17mYour AI Deserves Better Data
21mThe Exploratory Labeling Assistant: Mixed-Initiative Label Curation with Large Document Collections
50mSpeeding up data labeling with neural nets: pipeline, metrics and life hacks
09mEvalAssist: Insights on Task-Specific Evaluations and AI-Assisted Judgment Strategy Preferences
33mKeys to Data-Centric AI: Asking AI to Improve its own Dataset.
20mEmploying AI to extract actionable insights from qualitative user...
20mTowards rapid interactive machine learning: evaluating tradeoffs of classification without ...
26mBuild AI Systems for Discernment, Not Approval