We accidentally deleted production, but we made it not matter all that much
summary
A tiny change causes production to go away: deleting two deleted variables from an external secret operator triggers deployment automation, because Arguscd saw the secret gone for 10 milliseconds and interprets that as an undeploy. A subset of clusters actually un-deploy and the site is offline for 21 minutes across a few workload clusters before the secret returns and Argo redeploys; automation spreads both failures and fixes quickly, and the GitOps pipeline proves capable of fast recovery.
AI-written summary · about
related talks
37mEffective Disaster Recovery: The Day We Deleted Production
48mHow a Couple of Characters (and GitOps) Brought Down Our Site
27mWhen One Line Took Thousands of Websites Offline
22mOops, I deployed too hard
24mScaling GitOps Using ArgoCD Application-Sets: Lesson Learnt
26mManaging GitOps Deployments in Multi-cluster Production Environments
25mRUD: Rapid Unscheduled Deletion of Critical Infrastructure Workloads
14mHow to Not-Mess-Up Production