Did My Deploy Degrade Production?
summary
A production-ready script samples metrics every two minutes, trains on ~10,000 recent points and flags health changes by first cleaning data with KDE for major outliers (Silverman rule ± bandwidth tuning) and DBSCAN for minor blips (minPts=12, epsilon from elbow method), then computing percentile/σ thresholds to mark ailing vs unhealthy. It runs milliseconds per metric, removed clustered incidents and single-point noise, ran in customers’ production for years and worked reliably — adoption and monetization, not accuracy, are the failure modes.
AI-written summary · about
chapters (7)
- 0:00Introduction and problem
- 3:34Automated health detection
- 10:38Identifying outliers with KDE
- 21:18Automating outlier removal
- 33:26Setting health boundaries
- 38:01Customer adoption and future
- 40:30Q&A session
related talks
36mIt's not in Production Unless it's Monitored
36mThe ML Monitoring Flow for Models Deployed to Production
36mIt's Not in Production Unless it's Monitored
56mGetting all the 99.99(9) you always wanted
21mYour Deployments Are Lying. AI Knows
22mScaling Kernel Production Monitoring @ Meta
18mMaking Deployments Safe at Meta
19mAutomatic Metric Screening for Service Diagnosis