Dark Sky Camping: Reducing Alert Pollution with Modern Observability Practices
summary
Company scales from 100→1,000 parks (10x) in 18 months and faces alert chaos. Team moves from Prometheus+fragmented alerts into Honeycomb with the OpenTelemetry Java agent, funnels incidents through PagerDuty, and formalizes Service Level Objectives. Auto-instrumentation of core apps takes 4 hours to auto-instrument after ~150 hours of coordinated onboarding; developers add custom trace attributes like park_id and user_id. Outcome: far fewer noisy alerts, faster troubleshooting, and the ability to detect per-user slowdowns and decide whether an incident violates an SLO.
AI-written summary · about
chapters (7)
- 0:00Start
- 0:16Campfire Story
- 2:17Kristen Smith
- 2:48Dark Sky Parks
- 5:19Sky Glow
- 5:59The History of Camp Spot
- 11:13Funnel Everything into Honeycomb
related talks
27mBuilding a Scalable Monitoring System
38mDemocratizing Observability
05mLightning Talk: How the Observability Team at Spotify Radically Decreased On-Call Alerts
36mThe road to world-class observability
16mFrom monitoring to mastery: Building an observability-driven culture
31mAlerting for Distributed Systems—A Tale of Symptoms and Causes, Signals and Noise
27mWant to Solve Over-Monitoring...
40mFuzzy Lines: Aligning Teams to Monitor Your Application Ecosystem