Explainable Monitoring for Successful AI Deployments
summary
A monitoring system called Fiddler injects explainability into production models by computing drift with Jensen-Shannon divergence and attributing it via a drift impact heuristic that combines feature importance with drift. It surfaces a 10% accuracy dip tied to a range violation caused by a spurious lowercase us job_location value, lets labels join asynchronously for delayed outcomes, integrates with SageMaker and batch pipelines, runs on Kubernetes for horizontal scaling, and customers report debugging time falling from three days to under an hour.
AI-written summary · about
chapters (8)
- 0:00Start
- 2:37Deployed Al Systems are Error-Prone
- 5:27Al Risks - Data Drift
- 6:21Al Risks - Data Pipeline Issues
- 8:20Al Risks - Performance Decay
- 9:20Al Risks - Bias
- 11:27User Workflow
- 12:54Drill-down analysis
related talks
19mMinimize Risks and Accelerate MLOps with Model Performance Monitoring and Explainability
34mTraining & Monitoring AI - Drift Detection
21mZero Downtime ML Deployments
19mModel Performance Monitoring - A Practitioner's Perspective
31mMonitoring machine learning models in production
30mDetect machine learning (ML) model drift in production
42mTesting Machine Learning Modules in Production
18mReliable Maintenance of Machine Learning Models