Data doesn't lie — but it can mislead: How to ensure integrity of your Machine Learning applications
summary
A production entity linker is built by harvesting Wikipedia anchor links, encoding mention and candidate page text into embeddings, and scoring with a feed‑forward network; it achieves 79% F1 while a simple prior baseline hits 78.2% and an oracle on a pruned 14% candidate set sets an upper bound of 84%. Temporal updates (winner later disqualified) and low inter‑annotator agreement reveal labeling ambiguity; predicting on training data exposes structural annotation errors.
AI-written summary · about
chapters (12)
- 0:00Introduction and background
- 1:28Formula 1 news portal project
- 8:52Importance of domain experts
- 11:00Building entity linking model
- 16:17Baselines for performance context
- 21:33Correctness and ambiguity in data
- 27:10Inter-annotator agreement issues
- 33:27Handling single annotator projects
- 37:18Structural data errors detection
- 41:30Defining the right evaluation goal
- 42:52Practical checklist for ML projects
- 46:11Q&A session
related talks
39mHow to Avoid structural biases in evaluating your Machine Learning/NLP projects
38mMaking neural content meaningful and truthful relying on OpenNLP tools
30mActive Learning and Human-in-the-Loop Machine Learning
35mIndustrial Applications of Machine Learning in Search
35mImproving Credibility of ML Solutions
35mIntelligent Ruby: Getting Started with Machine Learning
28mHindsight bias: A machine learning story
34mScaling Trust