Common pitfalls leading to wrongly estimated model performance
summary
Common mistakes that make performance look better than it really is: using the validation set for model selection so validation becomes training and introduces selection bias; leaking future or target information via feature engineering such as target leakage via categorical encoding or unnoticed duplicates; violating IID with time drift so a random k‑fold lies and instead use time‑windowed cross‑validation or nested CV; measure how performance scales with training size and verify pipelines (scikit‑learn) by shuffling targets.
AI-written summary · about
chapters (17)
- 0:00Model performance evaluation
- 1:08Understanding generalization in machine learning
- 5:26Validation sets fail
- 7:56Separate test set
- 10:25Balancing data splits
- 11:28K-fold cross-validation
- 12:31Nested cross-validation
- 14:21IID assumption
- 15:46Time-series IID violations
- 17:15Time windowing validation
- 18:40Training size bias
- 19:02Underestimating with bias
- 20:46Duplicate records impact
- 21:49Target leakage
- 24:39Detecting leakage
- 26:03Validation simulates use
- 26:45Key takeaways
related talks
14mEarly Test Strategies in ML Model Development Phase
32mThe Dangers of Mindless Forecasting
08mCommon Mistakes in Performance testing
08mConsiderations for producing the best AI and Machine Learning models
53mUnleashing the Black Box: Testing Machine Learning Systems
30mHow to Properly Test ML Models & Data
24mYou Should Probably Be Doing Nested Cross-Validation
28mHindsight bias: A machine learning story