Annotating data the right way
summary
Decide by concrete outcomes: uses Prodigy as the annotation tool, measures label quality with Kappa score and targets roughly 80% agreement, and prefers starting with a few hundred hand-annotated examples before iterating. Works through active learning or regex filtering to surface rare classes, holds out a sequential test set for true evaluation, scales with internal annotators (chat support for guideline edge cases) and only uses crowdsourcing for simple tasks with heavy majority voting (six of seven) to denoise labels.
AI-written summary · about
chapters (11)
- 0:00Annotation best practices
- 2:34Deciding to annotate data
- 5:00Exploring data and labels
- 7:08Label boundaries
- 9:41Testing guidelines
- 11:47Choosing annotation tools
- 13:10Annotator agreement
- 15:39Scaling with annotators
- 18:07Active learning for imbalance
- 23:48External annotation services
- 28:23Key takeaways
related talks
36mGold standard data: lessons from the trenches
40mBuilding new NLP solutions with spaCy and Prodigy
30mEfficient data labelling with weak supervision
39mHow to Avoid structural biases in evaluating your Machine Learning/NLP projects
18mReality is not an end-to-end prediction problem
37mBest Practices for Building AI Datasets
39mThe practice of acquiring good labels
34mScaling Trust