Beyond Loss and Accuracy: Closing the Observability Gaps in AI Training with...
summary
TrainCheck is an open-source Python module that instruments PyTorch training via monkey-patching, records API traces and parameter hashes, and checks inferred runtime invariants with preconditions at every step. It learns relation templates from passing/failing examples, uses selective sampling and tensor hashing to limit overhead, and integrates with WandB/MLflow. On evaluated real bugs it detects 18 of 20 bugs from the first iteration, pinpoints root causes in 10 cases, and shows a <2% false positive rate while it records API traces and variable state.
AI-written summary · about
chapters (13)
- 0:00AI training observability gaps
- 1:19Silent training failure example
- 6:19Silent bugs evade monitoring
- 8:53Invariance-based training monitoring
- 11:44TrainCheck captures training semantics
- 13:29Infer invariants from traces
- 17:28Learned invariants benefits
- 19:13TrainCheck design and effectiveness
- 21:00Demo detecting Bloom bug
- 24:53Bloom gradient bug case study
- 27:44Frozen encoder bug case study
- 29:12Silent training failures takeaways
- 31:24Closing and TrainCheck resources
related talks
17mTraining with Confidence: Catching Silent Errors in Deep Learning Training with...
30mHow to Properly Test ML Models & Data
30mHow to Trust Your Deep Learning Code
34mTraining & Monitoring AI - Drift Detection
07mFinding Bugs in Deep Learning Programs
20mTaming AI Infrastructure Failures with Agentic Debugging
15mAdaCheck: An Adaptive Checkpointing System for Efficient LLM Training with Redundancy...
43mMachine Learning Testing Ecosystem of Python