How to Avoid structural biases in evaluating your Machine Learning/NLP projects
summary
Assembles concrete evaluation details: mines Wikipedia links, prunes the KB to 14% of concepts, 84% coverage, trains an entity linker with Spacey and scores ~79% F1, then compares that to a prior-probability baseline at 78.2% to show almost no gain; recommends annotation tooling like Prodigy, iterative prototype→production splits, document-level deterministic train/dev/test hashing, learning curves, extrinsic evaluations, and simple cascade classifiers — all focused on where the errors and label mismatches actually come from.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 2:55Zero-shot learning limitations
- 4:41Evaluating evaluation
- 8:54Prior probability bias
- 13:47Structural biases in data
- 17:14Annotation guidelines
- 22:48Inter-annotator agreement
- 25:58Extrinsic evaluation
- 27:24Corpus construction
- 31:54Input data selection
- 35:25Train-dev-test splitting
- 38:33Summary
related talks
54mData doesn't lie — but it can mislead: How to ensure integrity of your Machine Learning applications
34mBuilding Trust with LLMs
35mEnsuring Reliable Evaluation Systems for Your ML
29mAnnotating data the right way
26mHow can I trust you AI and the leap of faith
53mUnleashing the Black Box: Testing Machine Learning Systems
54mKeynote: Can you trust your (large language) model?
30mActive Learning and Human-in-the-Loop Machine Learning