Distributed Monitoring: How to Understand the Chaos
summary
Built a monitoring stack for a distributed microservice cloud that combines metrics, logs and traces and insists on putting the trace ID in logs so logs and spans join. Metrics arrive via Telegraf + Kapacitor + Chronograf; Kapacitor downsamples and forwards hot data, the cloud keeps a two-week retention window and currently stores about 310 distinct metrics. An OpenTracing-compatible tracer enriches logs and feeds Jaeger for visualization; lessons: watch cardinality, downsample aggressively, prefer UDP for high-volume metrics, and alert on aggregated signals not raw counters.
AI-written summary · about
chapters (5)
- 0:00Start
- 6:22The secret is all about how do you combine things together
- 8:04Visibility on 3rd party services We rely on a lot of them: Cloud, Mail
- 10:56There is not a unique recipe about how to instrument applications
- 26:19Fast iteration on how you collect metrics
related talks
42mDistributed Monitoring: How to understand the Chaos- 42mDistributed Monitoring: How to understand the Chaos
28mMonitoring Microservices w. Prometheus & Tracing Apps w. OpenTracing
49mImproving Observability of Distributed Systems
01h 02mAdding Observability to Distributed Systems
34mReduce Complexity with Distributing Tracing
59mDistributed Tracing and Monitoring with OpenCensus
42mFirst Steps with Observability: OpenTelemetry and You