Production Cluster Monitoring and Remediation for High Reliability
summary
eBay runs >13 Kubernetes clusters with ~8,000 nodes, 3,000 apps and ~100,000 pods and implements monitoring with Prometheus federation, filebeat logs into Kibana, custom metric exporters and assertion probes that simulate failures. Alerts feed Alertmanager with severity routing and automated incident tickets; dashboards jump from global to cluster to node views. A remediation controller uses a CRD-based node object, node problem detector inputs and the Reparo REST API to trigger repairs with change budgets; AIOps is still research but the automated flow is live.
AI-written summary · about
chapters (17)
- 0:00Intro
- 0:41Growing Clusters
- 1:31Monitoring Goals
- 2:16Monitoring Overview
- 3:08How we logging
- 7:07Assertion
- 7:41How we collect metrics
- 8:45How we build dashboards
- 10:01Global Health Dashboard
- 10:43Cluster Health Dashboard
- 11:10Node Health Dashboard
- 11:34How we manage alerts
- 14:08Global Alert Dashboard
- 16:33Monitoring Automation
- 22:08Overall
- 24:50Hardware Failures
- 26:16Software Failures
related talks
28mBuilding and Managing Kubernetes with Kubernetes
32mSurviving From Endless Issues Coming From 7K+ Kubernetes Clusters
42mMaking Resilient, Self-Adaptive, And Autonomous Kubernetes Clusters
29m“If you Don’t Monitor your Infrastructure, you Don’t Own it!” Regain Control Thanks to Prometheus
19mKeynote: Monitoring Kubernetes Clusters with Prometheus
41mScaling Reliability Engineering with Tools
35mImperative to Declarative Infrastructure by For Developers and Operators
24mTaming Observability at Scale in a Multi-Cluster Kubernetes Platform at Bloom...