Automating Kibana Alerting: When GitOps meets UI
summary
Company consolidates logs and metrics into Elastic and moves alerting to Kibana to tame 3 billion logs daily and 2,500 alerts per day across 1,300+ microservices. A GitOps+UI workflow uses YAML alert definitions in GitHub, Jenkins to run Terraform with the elasticsearch provider, and Python for migrations; Kibana audit logs plus webhooks generate PRs when alerts change. Features implemented include bi-directional automation (UI ↔ Git), maintenance-window realerting and spike detection, and a service factory that auto-creates alerts for new services.
AI-written summary · about
chapters (7)
- 0:00Introduction to the Journey of Observability
- 0:11Building Initial Observability
- 3:42Challenges of Scaling Observability
- 6:07Centralizing Logs and Metrics
- 12:12Implementing Bi-Directional Automation
- 18:17Advanced Alerting Features
- 23:06Future Plans and Conclusion
related talks
48mAlert Management as Code
25mStatuscake vs Kibana alerting
26mHow We Gained Observability Into Our CI/CD Pipeline
50mApplied Alerting Philosophy
27mAlert Fatigue to Action: Observability as the Backbone of Practical AIOps with OpenSearch
39mBeyond Logs: Achieving Observability with Selenium and Grafana
32mWho Are Your Alerts For?
48mDevOps Transformation: Introducing Incident Management and Maximizing Monitoring Value