How a Couple of Characters (and GitOps) Brought Down Our Site
summary
A single-line Helm template change causes Argo CD to interpret zero target clusters and Argo CD autoprune deleted 478 services across every cluster and region, dropping traffic to zero; reverting the PR restores namespaces but not deployments so manual restores and backups take hours. Recovery leverages a cells architecture served critical traffic from four clusters in one region, spot-instance cost savings, and runbook coordination. Concrete fixes: unit-test templating, channeled rollouts (dev/alpha/beta/main), disable dangerous auto-prune, and practice DR restores.
AI-written summary · about
chapters (5)
- 0:00Start
- 23:22The greatest risk comes whilst crossing the chasm
- 24:04Templates with logic are code
- 24:37Don't do global config deploys
- 25:03A true blameless incident culture is invaluable
related talks
26mGitOps Me Some of That! Managing Hundreds of Clusters with Argo CD
05mWe accidentally deleted production, but we made it not matter all that much
27mArgoCD as the Engine of a Brownfield Migration to Kubernetes
21mSet It To Auto: Using Argo CD Autopilot To Simplify Everything
25mRUD: Rapid Unscheduled Deletion of Critical Infrastructure Workloads
24mScaling GitOps Using ArgoCD Application-Sets: Lesson Learnt
27mUps, let's revert! 🔥 Automating Kubernetes Rollouts with Argo a...
29mTuning Argo Rollouts for Thousands of Workloads