More like this: monitoring recommender systems in production
summary
Practical checklist: measure a revenue KPI first, add real‑time proxy metrics (clicks, views, conversion) and batch compute offline metrics like Precision@K and NDCG every 10 minutes, hourly or daily; track novelty, diversity and popularity bias; monitor service health and input data quality/drift; create synthetic avatar users and use item/user‑level case tables for debugging; use Evidently reports (including the RecCase table) to compare reference vs current batches and alert on surprising shifts so root cause is actionable.
AI-written summary · about
chapters (10)
- 0:00Monitoring recommender systems
- 3:36Business metrics limits
- 6:27Online quality proxies
- 8:13Offline model metrics
- 10:39Accuracy and beyond
- 14:55Debugging production issues
- 19:11Balancing monitoring and debugging
- 20:15ML observability framework
- 23:07Summary
- 24:30Q&A
related talks
41mRecommender Systems: Crafting Effective Personalised Experiences
38mRecommendations at Rappi: from MVP to Data Product
40mDesigning Data Driven Projects: Beyond ML Experimentation
37mWhat I learned from monitoring more than 30 Machine Learning Use Cases
43mRecommendation Systems in Market Research
36mApplied Machine Learning Playbook: From Idea to Production
30mFrom €1M License to In-House Success: How We Built a Real-Time Recommendation System and Saved Millions Doing It
18mReliable Maintenance of Machine Learning Models