Lightning Talk: Why Logging Isn’t Enough: Making PyTorch Training Regressions Vi...
summary
PyTorch training regressions become visible by turning logs into operational checks: scheduled short and long baseline runs compared to a 30‑day average, standardized metrics for throughput, memory, loss, divergence, and simple statistical baselines that trigger alerts when a run exceeds historical stddev. Implementation uses an in‑training background thread to log a global step heartbeat and cancel jobs stalled >1 hour, plus post‑run comparisons tied to Git to find the commit that introduced cost or quality regressions. Tradeoffs: false positives, alert fatigue, baseline drift.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 1:25Healthy training
- 2:35How regressions manifest in practice
- 3:39Catching regressions
- 5:26Two monitoring lenses
- 7:09Training monitoring architecture
- 8:11Wave examples
- 10:38Why now
- 11:20Key takeaways
related talks
27mLogs Are Good, Actually
05mLightning Talks from Robson Port
18mReliable Maintenance of Machine Learning Models
34mLightning Talks
27mSave your ML model from degradation with data logging
15mWhy I Prefer Good Testing Over Excellent Testing
58mLightning Talks 2
06mLightning Talk: Automated Testing Done Right