Productionizing Machine-Learning Services: Lessons from Google SRE
summary
Claim: models must be retrained continuously because accuracy decays in days or weeks; Google sees production models age fast and YouTube’s “watch next” needs constant snapshots and re-labeling to avoid viral bias. Training is heavy (models >1 TB, GPUs→TPUs cut hours to minutes) and needs resource isolation; serving needs ~1 ms or heuristic fallback. Operational rules include continuous retraining, signed models and canaries, data deduplication and imputation, domain-specific alerts, and GDPR deletion automation.
AI-written summary · about
chapters (14)
- 0:00Intro
- 1:51What is ML good for?
- 4:01Youtube ML for video recommendations
- 8:31Theoretical Machine Learning Pipeline
- 11:57Training: Filtering is key
- 14:06Training: Data size
- 14:54Training ot Scale
- 16:26Summary: Data Quality on Training
- 20:25Why hardware resources are important
- 22:20Model Qualification
- 24:13Backwards Compatibility
- 26:50Rollbacks and Cloning
- 32:29Privacy: When using an individual's data
- 33:34Ethics in ML
related talks
38mBuilding robust machine learning systems
18mReliable Maintenance of Machine Learning Models
39mAn SRE Approach to Monitoring ML in Production
30mFrom Black Box to a Known Quantity: How to Build Predictable, Reliable ML-based
01h 05mMachine Learning Models and Productionization
42mImproving Machine Learning Development Reliability
44mSRE for ML: The First 10 Years and the Next 10
28mWhy we do ML model retraining wrong, and how to do better