Training & Monitoring AI - Drift Detection
summary
Argues that deployed models need ongoing statistical checks, shows classical two-sample tests fail on multimodal shifts and that partial matching of distributions over latent features catches subtle regime changes; warns that running tests “1000 times a day” forces p-values far below standard 5% or else thousands of false alarms; demonstrates an implementation in open source TorchDrift and the service DriftDash that runs on-prem or edge, and reports it successfully surfaces real drifts without constant false positives.
AI-written summary · about
chapters (7)
- 0:00Intro
- 5:12Agenda
- 5:53Why detect drift?
- 12:51Statistics of detecting drift
- 21:21Practical drift detection
- 32:02Conclusion
- 33:52Outro
related talks
30mDetecting drift: how to evaluate and explore data drift in machine learning systems
18mReliable Maintenance of Machine Learning Models
25mMonitoring AI with AI
31mMonitoring machine learning models in production
26mProduction Data to the Model: “Are You Getting My Drift?”
45mExplainable Monitoring for Successful AI Deployments
36mThe ML Monitoring Flow for Models Deployed to Production
37mAI/ML Strategies & Techniques for a Post Pandemic World