Lightning Talk: How the Observability Team at Spotify Radically Decreased On-Call Alerts
Lightningat KubeCon + CloudNativeCon North America 2019 · on Tuesday, 2019-11-19· at 17:38· Exhibit Hall AB - San Diego Convention Center, Ground Level·
summary
Reliability team takes over legacy monitoring and within six months migrated to Prometheus and Grafana, moved metric forwarding to FFWD and storage to Heroic, and shifted workloads to Kubernetes and a managed pub/sub, which removes broker ops and adds autoscaling; custom almond alerting integrates with Griffin and remains open source; alerts drop from 76 to ~6 and on‑call pages fall dramatically (uploader reports 42% initially, internal recheck cites 92%), freeing time for experimentation.
AI-written summary · about
chapters (7)
- 0:00Intro
- 0:44Goals
- 1:11Lowhanging Fruit
- 1:42Pinger
- 2:08Monitoring Stack
- 2:35Custom Solutions
- 3:54Operational Load
related talks
12mLightning Talk: Faster Insights and Improved Accuracy: Spotify's Prometheus Upgrade
13mLightning Talk: Observability-First DevSecOps: Building Resilient Multi-Cloud P...
29mParanoid Observability –Success Stories and Misfortunes observing an observability stack
29mObservability; a gentle introduction
29mDark Sky Camping: Reducing Alert Pollution with Modern Observability Practices
28mMonitoring Stack was yesterday – Observability Platform at scale!
56mFull-stack observability and application monitoring with
52mBuilding a culture of observability