Don't page me! How we limit pager noise at New Relic
summary
A large New Relic monolith handles 200k req/min across more than 40 external services and 11 MySQL databases and yet paging drops from about five times a week to once a month by doing specific work: retro every notification, record which alert condition fired and how the incident ended, refine default alerts with NRQL and custom attributes (customer name, user agent), run heartbeat monitors / synthetics as continuous integration-style checks, and iterate until alerts reliably match real customer impact.
AI-written summary · about
chapters (6)
- 0:00Start
- 3:403 point takeaway
- 7:42Pre-built alert conditions
- 15:16Gradual refinement of alert conditions
- 26:57Heartbeats / Synthetic monitors
- 34:21Use Semian's bulkheads
related talks
27mBuilding a Scalable Monitoring System
11mBuilding a system that never stops: New Relic at scale
19mSensory Friendly Monitoring–Keep the Noise Down
36m"Alert overload: How to adopt a microservices architecture..."
27mWant to Solve Over-Monitoring...
32mAre We All on the Same Page? Let's Fix That
38mEffective App Monitoring in Production
23mMastering Monitoring & Alerting: Key to Seamless Performance