Minder: Faulty Machine Detection for Large-scale Distributed Model Training
summary
Minder is an automated faulty-machine detector deployed for over a year on thousand‑machine distributed training jobs; it builds per‑metric parametric models using a VAE with LSTM encoder/decoder, compares per‑machine reconstruction scores for machine‑level similarity and continuity, and flags machines that show consecutive anomalous windows. On production traces and PCIe‑downgrade injections it locates roots quickly, reducing detection time by 99% and reacting in 3.6 seconds on average with precision 0.904 and F1 0.893.
AI-written summary · about
chapters (10)
- 0:00Minder introduction
- 1:19PCIe downgrading example
- 2:22Manual fault challenges
- 3:27Fault occurrences analysis
- 4:12Fault types and anomalies
- 6:01Consistent behavior insights
- 7:07Minder system design
- 9:33Minder deployment results
- 12:05Concurrent faults impact
- 13:28Production insights
related talks
21mEnhancing Runtime Reliability in LLM Training via Fine-Grained Observability
31mMonitoring machine learning models in production
42mTesting Machine Learning Modules in Production
18mReliable Maintenance of Machine Learning Models
33mAt the age of AI and ML
51mKeynote: Fault Tolerant Machine Learning Operations
25mMonitoring AI with AI
19mModel Performance Monitoring - A Practitioner's Perspective