The Exploratory Labeling Assistant: Mixed-Initiative Label Curation with Large Document Collections
summary
A mixed-initiative system called ELA supports label ideation, specification and evaluation over large text collections using a three-panel UI (Discovery, Specification, Evaluation) and keyword-based rules with AND/OR/NOT and projection views. It runs interactive queries on 20k–1M documents in real time, achieves 0.94 precision for user-created specifications on scam detection, discovers 90 new scam types in an Agari case study, scores excellent SUS in a user study on 19k Yelp reviews, and SVM retraining on user labels raises recall while keeping precision.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 0:30Problem Statement
- 3:23Property Labeling
- 4:26User Profile
- 4:38Example
- 8:00Demo
- 8:21Agari Example
- 9:21User Study
- 10:28User Focus
- 11:47Discovery
- 12:23Evaluation
- 13:47Results
- 15:55Conclusion
related talks
12mIncreasing the Speed and Accuracy of Data Labeling Through an AI Assisted Interface
01h 26mShipping AI That Works: An Evaluation Framework for PMs
10mSteering Semantic Data Processing With DocWrangler
29mAnnotating data the right way
09mEvalAssist: Insights on Task-Specific Evaluations and AI-Assisted Judgment Strategy Preferences
56mDehumanizing Agents: Why Explainability is Crucial in the LLM Era
22mAgentic Hybrid Retrieval: Orchestrating LLM Agents, Hybrid Search & Explainability in OpenSearch
17mFrom messy to meaningful data: LLM-powered classification in R