Training with Confidence: Catching Silent Errors in Deep Learning Training with...
summary
TRAINCHECK automatically infers low‑level training invariants from reference traces and uses selective dynamic instrumentation (monkey‑patching and proxies) to verify them at runtime, flagging violations within iterations. It reproduces 20 real silent errors and detects 18 of 20 within a single iteration, pinpoints root causes in 10 cases, uncovers 6 new bugs in popular libraries, keeps false positives below 2% across 63 correct pipelines, and adds under 11% overhead with 100 invariants deployed.
AI-written summary · about
chapters (11)
- 0:00Silent training errors intro
- 2:44Empirical study of silent errors
- 5:12Bloom divergence root cause
- 7:19Train Check overview
- 8:24Challenges in invariant inference
- 10:08Automated invariant inference
- 12:18Role of preconditions
- 13:21Dynamic instrumentation
- 14:04Evaluation on real errors
- 15:49Generalization and overhead
- 16:32Summary and open-source
related talks
33mBeyond Loss and Accuracy: Closing the Observability Gaps in AI Training with...
07mFinding Bugs in Deep Learning Programs
24mTesting Techniques for Critical Deep Learning Applications
15mStatic Detection of Silent Misconfigurations with Deep Interaction Analysis
30mHow to Properly Test ML Models & Data
30mHow to Trust Your Deep Learning Code
25mDeepBugs - A Learning Approach to Name-based Bug Detection
34mTraining & Monitoring AI - Drift Detection