Testing Machine Learning Systems: What it is and why you should care
summary
A real project builds a smart-quotation pipeline for custom automotive parts that chains OCR, NER, feature extraction and several models (visibility, step count, full production flow) into a web app; preprocessing functions like resizing, color-to-BW and untilting get unit tests for preprocessing, data checks assert schema, label presence, distribution and leakage, models get invariance and directional expectation tests plus minimal example checks and seeded reproducibility, and production uses end-to-end production monitoring — result: two-year contract showed near-zero maintenance time after testing.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:50Why testing ML matters
- 3:14Challenges in ML testing
- 7:27ML development phases
- 9:11Testing data engineering
- 12:25Evaluating ML models
- 14:31Monitoring ML in production
- 17:45Integrating models
- 18:46Post-deployment challenges
- 20:41Q&A
related talks
31mTesting Machine Learning Models
53mUnleashing the Black Box: Testing Machine Learning Systems
25mFuture of Software Testing: Machine Learning Assistance
38mBuilding robust machine learning systems
24mMachine Intelligence gets real
30mHow to Properly Test ML Models & Data
38mNext Generation Testing in the Age of Machines
28mML-Driven Test Intelligence at Scale - What Works What Fails and Why It Matters