The practice of acquiring good labels
summary
Focuses on concrete methods: program crowds with clear instructions and short hits, use two-phase filtering (cheap Mechanical Turk binary filters then higher-skilled verification) to scale near-duplicate and content tasks, debug tasks by iterating prototypes and sampling, and apply adaptive stopping rules so easy items use few workers while hard items get more. Recommends majority vote when history is scarce, EM when worker history exists, and behavioral fingerprinting (mouse/timing) to boost quality alongside gold sets and redundancy.
AI-written summary · about
chapters (7)
- 0:00Introduction to label quality
- 2:46Why labels matter in ML
- 10:17Wet programming for humans
- 19:57Testing and debugging tasks
- 27:36Quality control strategies
- 33:12Key takeaways and summary
- 35:39Q&A on label acquisition
related talks
29mGotta catch'em all recognizing sloppy work in crowdsourcing tasks
36mGold standard data: lessons from the trenches
29mAnnotating data the right way
25mGotta catch'em all: recognizing sloppy work in crowdsourcing tasks
20mGreenfield vs. Brownfield Data Labeling
36mApplied Machine Learning Playbook: From Idea to Production
42mClean Labels for ML
23mLearning with Limited Labels & Weak Supervision