Real-Time Monitoring of Distributed Systems
summary
Trademob measures millions of time-series points from over +700 instances and spikes to 125k requests per second and builds a Python-based real-time monitoring system that correlates metrics, reduces false positives, and triggers actions (restart or kill instances). Algorithms include interquartile outlier filters across groups, seasonal-trend decomposition for temporal anomalies, and a memory‑efficient t-digest for extreme quantile thresholds; the pipeline runs near‑real‑time and proves effective at lowering alert noise and finding causal signals.
AI-written summary · about
chapters (10)
- 0:00Real-time monitoring intro
- 3:22Anomaly detection limits
- 5:52Building Anom for scale
- 8:20Metric pipeline
- 10:08Anomaly patterns
- 11:55Outlier detection
- 14:45System performance analysis
- 19:03Dynamic thresholds with T-digest
- 25:53Open-sourcing Anom
- 26:40Future of monitoring
related talks
18mReal-Time Monitoring of Distributed Systems
17mRealtime Monitoring
12mModel Monitoring: Detecting and Analyzing Data Issues
33mDistributed Tracing - All The Warning Signs Were Out There!
52mReal-time-first metrics
26mML Monitoring & Observability
41mTracing Real-Time Distributed Systems
26mDistributed Tracing: All the Warning Signs Were Out There!!!