Model Upgrade Schemes- Considerations for Updating Production Models
summary
Describes an implemented CI/CD + continuous monitoring and training pipeline with feature store integration, a random holdout feedback loop and delayed labels, and compares models with frequentist tests (paired t, correlated t, 5x2 CV) versus a Bayesian correlated test that returns posterior probabilities and a region of practical equivalence. Gives dataset sizes (~70k records), reports cases where Bayesian rope shows ~60% equivalence vs ~40% win, discusses tradeoffs in runtime, interpretability and when retraining actually improves business metrics.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 3:15Problem statement
- 7:31MLOps deployment limits
- 10:59Four Cs of MLOps
- 16:53Model validation
- 20:05Frequentist tests
- 24:15Bayesian validation
- 33:44Insurance model considerations
- 37:40Q&A
related talks
29mAutomated Model Update & Evaluation
27mI have no clue what my model is doing...
27mModel Evaluation for Humans
30mFrom Black Box to a Known Quantity: How to Build Predictable, Reliable ML-based
38mBuilding robust machine learning systems
36mThe ML Monitoring Flow for Models Deployed to Production
33mDealing with the versioning of production-ready models
46mProduction Model Deployment