A simple way to validate and monitor the performance of your ML Applications.
summary
A concrete recipe for keeping models healthy: run a small validation dataset anchored to model inside the app (example target 100 predictions in 10 seconds with ~80–82% accuracy and F1 ~80–81), record accuracy F1 duration and confidence, run the check daily via fastapi repeat decorator on EC2, and flag outliers. Low‑confidence cases go to a secondary stronger model or human review. Real outcomes include catching a numpy/dependency change and hardware/VM regressions.
AI-written summary · about
chapters (10)
- 0:00Start
- 3:40Performance Changes
- 6:36Data Drift/Covariate Shift
- 11:52Checking Model Performance
- 12:45Catch the early signs
- 14:48Overview
- 17:04The Setup
- 25:58Model Stability
- 26:50Model Longevity
- 27:18Cost of Maintenance
related talks
30mHow to Properly Test ML Models & Data
18mReliable Maintenance of Machine Learning Models
30mFrom Black Box to a Known Quantity: How to Build Predictable, Reliable ML-based
19mApples, Oranges, and ML Models: Model Validation vs Benchmarking
27mModel Evaluation for Humans
19mModel Performance Monitoring - A Practitioner's Perspective
30mOn scaling Machine Learning microservices
39mAn SRE Approach to Monitoring ML in Production