RUD: Rapid Unscheduled Deletion of Critical Infrastructure Workloads
summary
PostFinance runs 30 vanilla Kubernetes clusters and deploys infra stacks with ArgoCD ApplicationSets. A force-replace sync causes ArgoCD to start deleting namespaces, which removes the metrics service backing the kube‑metrics APIService and makes the API unreachable. Kubernetes cannot finish namespace deletion while that APIService is down, leaving many namespaces stuck terminating and observability blind. Recovery is to delete the APIService object to break the circular dependency. Safeguards added: split ArgoCD instances and preserveResources on deletion.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:50Agenda and infrastructure
- 4:20Defining workloads
- 5:46Argo CD Application Sets
- 8:36November 2023 incident
- 10:00Root cause analysis
- 11:25API Services and dependencies
- 14:15Breaking dependencies
- 14:58Safeguards implemented
- 16:00Other dependencies
- 17:25Live incident demo
- 20:57Q&A
related talks
48mHow a Couple of Characters (and GitOps) Brought Down Our Site
47mPostmortem: A Modern Approach to Managing Kubernetes Clusters
27mArgoCD as the Engine of a Brownfield Migration to Kubernetes
37mEffective Disaster Recovery: The Day We Deleted Production
09mLightning Talk: How to Avoid a Kubernetes Doom Loop
31mHow the OOM-Killer Deleted My Namespace, and Other Kubernetes Tales
21mHow to Not Destroy Your Production Kubernetes Clusters
24mWhoops! How Not to Accidentally Delete Everything.