Lessons Learned Building a Global Synthetic Monitoring System
summary
Built a global synthetic monitoring network using blackbox exporter, deploying 22 probe locations across 9 Grafana Cloud regions (≈198 probes total) on self‑managed Kubernetes via RKE. Design uses Prometheus metrics, per‑location canaries and staggered rollouts driven by Argo workflows, Calico network policies to deny private ranges and enforce IPv4/IPv6 controls, and published probe IPs and verbose user‑agents to avoid bot blocks. The system surfaces DNS/IPv6/capacity blind spots, stops bad releases automatically, and materially reduces false alerts.
AI-written summary · about
chapters (14)
- 0:00Intro
- 0:36Poll: Are you familiar with Synthetic Monitoring?
- 0:54What is Synthetic Monitoring?
- 1:45Why do we need Synthetic Monitoring?
- 7:34Why Blackbox Exporter?
- 9:17Why do we need a global network?
- 17:30VMs: simple and easy
- 25:50Agent release rollouts
- 29:36Parts of Internet will always be unreliable
- 30:05Add Redundancy where you can
- 30:32Maintenance Windows
- 31:22Good Defaults & User Education
- 31:58Limits and Abuse
- 34:44Implicit Dependencies
related talks
35mMonitoring Distributed Java Applications at Scale
23mDon't Let Users Find Your Outages: Synthetic Monitoring for Kubernetes Platforms
24mPractical Kubernetes Monitoring with Prometheus
19mKeynote: Monitoring Kubernetes Clusters with Prometheus
24mTaming Observability at Scale in a Multi-Cluster Kubernetes Platform at Bloom...
32mAdapting Brownfield Deployments to Cloud-Native Technology
50mMetrics and alerting with Prometheus and Grafana
35mFrom New Cluster to Insight. Deploying Monitoring and Logging to Kubernetes