Auto-remediation at scale using watchers
summary
Shows concrete Watcher plugin for Elasticsearch watch definitions and the exact log queries used to detect failures from application and infrastructure logs, plus the action types sent (email, webhook, index) and how those map to automation scripts. Demonstrates an event-driven automation flow that triggers remediation commands, confirms service restart, and logs the outcome so alerts stop. Covers limitations around rate and stateful retries and the tradeoffs of alerting and notification versus automated changes to self-heal failed systems.
AI-written summary · about
chapters (5)
- 0:00Auto-remediation at scale using watchers
- 1:54Problem Discovery?
- 3:39What's monitoring at scale?
- 8:12Scripting is costly!
- 19:41Design a watch in ES to self-heal
related talks
26mWho Watches the Watchers? Building Observability for the Platform Itself Across Multi-Cluster EKS
20mBridging the Safety Gap from Scripts to Full Auto-Remediation
36mEvent-based self-healing monitoring
23mAutomated Resilience: Using Argo Events for Real-Time Incident Remediation
08mSponsored Keynote: The Work Before the Magic: Autoremediation Readiness
50mAutomating Kubernetes with Event Driven Ansible
14mFrom Dashboard to Defense: Automating Resilience at
17mMonitoring as Code