Real-Time Monitoring of Distributed Systems
summary
Trademob runs up to 125k requests/sec across 700+ instances and was drowning in 2k–5k metrics/sec, so engineers build Anam, a Python pipeline that multiplexes tagged metrics into multiple granular series, stores them in Redis, and dispatches analysis jobs via Celery/libuv. Detection combines interquartile range outlier filters across instances, seasonal trend decomposition per time series to model expected behavior, and online quantiles via T-digest for dynamic thresholds, then auto-restarts or kills boxes or emits alerts.
AI-written summary · about
chapters (6)
- 0:00Introduction to monitoring
- 2:58Building Anomal detection system
- 8:21Algorithm overview and examples
- 10:31Detecting system behavior anomalies
- 13:45Seasonal trend decomposition
- 17:02Open sourcing Anomal
related talks
29mReal-Time Monitoring of Distributed Systems
52mReal-time-first metrics
35mBuilding a Real Time Metrics Database
33mNetdata: Open Source, Distributed Observability Pipeline – Journey and Challenges
12mModel Monitoring: Detecting and Analyzing Data Issues
33mDistributed Tracing - All The Warning Signs Were Out There!
26mDistributed Tracing: All the Warning Signs Were Out There!!!
22mEnd-2-End Monitoring and Troubleshooting a Real-Time Data Pipeline