Changing the engine in flight: rebuilding monitoring at scale
summary
Rebuilds a live monitoring stack piece by piece to replace a polling datacenter agent with a pushable metrics pipeline using Go-based microservices, an ambassador API (HTTP+JSON + swagger), service discovery and dynamic config, and a global backend of Kafka with ElasticSearch feeding Grafana/Kibana. A real incident — one service suddenly emitting 500 metrics per machine — exposes scaling limits, so the new collector, agent-as-library and global store work but leave a hybrid system and deprecation work still ongoing.
AI-written summary · about
chapters (6)
- 0:00Start
- 5:48APPLICATION MONITORING
- 13:08MICROSERVICE ECOSYSTEM
- 14:42KEY INGREDIENTS
- 27:18CHANGING THE BACKEND
- 30:47CAVEATS
related talks
10mMonitoring Microservices: A Challenge
27mMonitoring as Code: Managing your Observability Assets the Cloud-Native Way
28mMonitoring Stack was yesterday – Observability Platform at scale!
45mDecentralizing SaaS Monitoring Using Open-Source Tools and Standards
25mAdopting Open Telemetry Collector @ eBay - Swapping Engines Mid Flight
29m“If you Don’t Monitor your Infrastructure, you Don’t Own it!” Regain Control Thanks to Prometheus
11mBuilding lightweight observability system for Lean Teams
35mMonitoring Microservices