How We Discover and Locate k8s Cluster Problems Before Users at Alibaba
Talkat KubeCon + CloudNativeCon + Open Source Summit China 2021 · on Thursday, 2021-12-09· at 12:10· Kubecon + CloudNativeCon 演讲厅, 虚拟会议 (Virtual)·
summary
Describes Alibaba’s in-house tool KubeProbe and related practices for early detection and localization of problems in large-scale Kubernetes clusters. Covers a universal link-detection approach that simulates broad user behaviors, targeted inspections of cluster metrics to spot current or future risks, system enhancements for faster root-cause analysis, and post-detection actions including checks, self-healing and Chat-Ops.
AI-written summary · about
chapters (13)
- 0:00Intro
- 0:42Agenda
- 1:20Background
- 5:34Challenges
- 7:05Thoughts
- 11:02Solution
- 12:08Link Detection
- 14:46Targeted Inspection
- 15:23System Implementation
- 19:44After Problem Discovery
- 22:58Root Cause Analysis and Post-Processing
- 25:33Chat-ops
- 26:47Example Demo
related talks
35mNo More Chaos: Audit and Inspect Kubernetes at Scale
35mNo More Chaos: Audit and Inspect Kubernetes at Scale
18mK8s Troubleshooting Demystified: Five Best Practices
37mProactive Kubernetes Security: Anomaly Detection and Runtime Alertin...
32mSurviving From Endless Issues Coming From 7K+ Kubernetes Clusters
33mAm I Using It Right? Checking Best Practices on Live Kubernetes Clusters
40mThe Good, the Bad, and the Ugly of Managing Production Systems with Kubernetes
29mProduction Cluster Monitoring and Remediation for High Reliability