ML Monitoring & Observability
summary
Meta diagnoses spikes in normalized entropy (any) explosions by correlating training logs, feature stats and lineage, and automates fixes like hyperparameter rollbacks, feature blocking for NaN bursts, gradient clipping hints and platform-version bisection. Platform pieces include a lightweight+heavy statistics pipeline, traceability lineage for upstream root cause, centralized storage/query and low-overhead fine-grained instrumentation for checkpoint analysis and causal queries. Goal is near-real-time detection and reduced MTTD/MTTR across large models and datasets.
AI-written summary · about
chapters (5)
- 0:00ML monitoring intro
- 3:56Exploding gradients explained
- 9:56ML debugging platforms
- 20:27Building monitoring platform
- 23:01Meta’s ML monitoring story
related talks
12mModel Monitoring: Detecting and Analyzing Data Issues
10mML-based Log Analysis for Faster Debugging
33mBeyond Monitoring: The Rise of Observability Platform
42mMachine Learning In Observability Systems: From Insights To Action
01h 21mBuilding a ML Model Observability pipeline
30mBetter data quality with ML & visualization
21mAnomaly Detection for Platform Logs using Embedded Context
19mModel Performance Monitoring - A Practitioner's Perspective