Who Watches the Watchers? Building Observability for the Platform Itself Across Multi-Cluster EKS
summary
Platform observability for 11 EKS clusters across 6 AWS accounts uses labeled alerts (service, namespace, owner team, owner channel) plus trace IDs to link metrics, logs and traces; an Alertmanager webhook runs machine-readable runbooks that execute read-only diagnostics, post root-cause to Slack, and surface PRs; the small agent is ~100 lines, prevents noisy paging, and converts incidents into runbooks so MTTR drops and on-call wakeups fall dramatically.
AI-written summary · about
chapters (19)
- 0:00Introduction
- 1:06Alert cycle inefficiencies
- 3:13Alerts must answer three questions
- 4:38Labeling alerts for ownership
- 5:41Unifying observability with labels
- 7:53Trace IDs and consistent labels
- 9:20Runbooks in incident response
- 10:24Runbook structure and purpose
- 12:30Machine-readable runbooks
- 13:34Triggering runbooks via webhooks
- 14:55Alerts flow to humans and agents
- 16:40Simplifying incident resolution
- 17:22Agent workflow steps
- 20:29Reducing mean time to resolve incidents
- 21:32Handling unmatched alerts
- 22:55Every incident to a runbook
- 23:16Guardrails for automation
- 24:02Watching the watchers
- 25:25Q&A
related talks
31mBest Practices for Effective Multi-Cluster Observability: Lessons Learned
24mObservability with AWS
56mFull-stack observability and application monitoring with
29mParanoid Observability –Success Stories and Misfortunes observing an observability stack
20mDesigning the Right Observability Stack in AWS Cloud
29mWho Watches the Watchers? From Closed Observability To...
01h 02mDesigning an open-source observability stack for Kubernetes
28mMonitoring Stack was yesterday – Observability Platform at scale!