From Black Box to a Known Quantity: How to Build Predictable, Reliable ML-based
summary
Sixty interviews across production teams yield five concrete rules: monitor model freshness because model quality degrades with age, stamp models with signatures to guarantee API/model compatibility, use canary deployments for incremental rollouts and automatic rollback, track resources separately for training versus serving (training is ~10% of effort but serving becomes the bottleneck), and enforce data filtering/imputation plus validation splits (70/30 or 80/20) with checkpoints so models can be retrained continuously and deployed every few hours.
AI-written summary · about
chapters (24)
- 0:00Intro
- 1:34Goals
- 2:30Black Box
- 3:15YouTube Watch Next
- 4:44ML Systems are Not Just Another Data Pipeline
- 5:49Theoretical ML Pipeline
- 7:15Best Practices
- 8:30Retraining
- 8:59Model Loss
- 9:29Illustration
- 11:15Model evolution
- 11:30Filtering
- 13:12Training
- 14:07Summary
- 16:38Resources
- 17:49Qualification
- 19:04Canaries
- 20:08Deployments
- 20:23Backwards Compatibility
- 22:45ML Pipeline
- 27:28User Happiness
- 28:21Conclusion
- 29:04Open Source Data
- 30:01Thank you
related talks
18mReliable Maintenance of Machine Learning Models
31mReliable Data for Large ML Models: Principles and Practices
39mAn SRE Approach to Monitoring ML in Production
08mConsiderations for producing the best AI and Machine Learning models
28mWhy we do ML model retraining wrong, and how to do better
30mHow to Properly Test ML Models & Data
30mWhen Good Models Go Bad: The damage caused by wayward models and how to prevent it
53mUnleashing the Black Box: Testing Machine Learning Systems