Lessons Learned in Black Box Monitoring 25,000 Endpoints and Proving the SRE Team's Value
summary
Built a black‑box monitor for 25,000+ .gov domains that records HTTP status and latency, starting with a Python GET-based MVP and scaling to Prometheus + blackbox exporter → AWS Lambda/API Gateway for cost and IP distribution reasons. Scrape intervals, timeouts and alert tuning prove key; run into Amazon abuse when probes hit too often, prompting move from EC2 proxies that cost $700 → $70/month. Result: long‑term Influx metrics, Grafana drilldowns, targeted alerts, and proof points to engage agencies.
AI-written summary · about
chapters (21)
- 0:00Introduction
- 0:42The Digital Service
- 3:33How do we find the problems
- 4:20The big solution so far
- 5:11Air Nowgov
- 5:52USPTO
- 6:39Black Box Monitoring
- 7:16Government Services
- 8:59MVP
- 9:43Architecture
- 9:59Alert Manager
- 11:16What are we monitoring
- 13:26Prometheus Black Box Exporter
- 13:40Time Series Data
- 14:57Amazon Threat
- 17:18API Gateway
- 18:07Whats Next
- 18:45Results
- 19:20Chart
- 19:54Lessons Learned
- 21:13Alerting
related talks
12mUnlocking the Secrets of Client Data
31mMastering Near-Real-Time Telemetry and Big Data: Invaluable Superpowers for Ordinary SREs
20mHow We are Dealing with Metrics at Scale on GitLab.com
22mOpen Telemetry for API Monitoring
16mShrinking the Observability Bill
31mPrecision Over Proliferation: SRE Approach for Leaner, Smarter and Data-Driven..
19mUser-centered observability: load testing, real user monitoring & synthetics
28mHow to reduce expenses on monitoring with VictoriaMetrics