Low-Resource Languages as Stress Tests for NLP Data
summary
Low-resource languages expose where annotation and evaluation collapse real ambiguity into a single label: examples like old friend and Oge’s surface form E ho ho e swe each support multiple grammatical readings so a benchmark that forces one ground truth misrepresents the data. Run three diagnostics: label stability (persistent annotator disagreement signals underspecification), guideline robustness (categories break across dialects and borrowings), and a numbers test that shows scores falling from 91% to 67% when you require discourse-aware polarity. Document theory, allow underspecified categories, and separate cue detection from semantic understanding.
AI-written summary · about
chapters (7)
- 0:00Introduction and claim
- 2:56Low-resource languages reveal assumptions
- 6:08Example from Oge language
- 8:37Three stress tests for NLP data
- 13:12Documentation and annotation tips
- 14:15Conclusion and call to action
- 15:58Q&A
related talks
46mContext is all you need: FUNdamental linguistics for NLP
01h 00mHuman and machine inductive biases for compositional linguistic generalization
31mRemaining Challenges in Deep Learning based NLP
36mGold standard data: lessons from the trenches
31mChallenges and Considerations in Language Model Evaluation
45mHow multilingual is your NLP model?
30mEfficient data labelling with weak supervision
55mLessons Learned From Implementing Generative AI Solutions