Touch Base: How I Broke Business Applications Deployment
summary
A Kustomize overlay mistake replaces a multi-line config and removes the ArgoCD routes ignoreDifferences entry, causing many apps and routes across clusters to go out of sync minutes after merge. Recovery is a revert; root cause is that multi-line YAML is replaced by patch, not merged. Response builds a merge-request pipeline that runs kubectl and ArgoCD dry-runs, enforces approvals for base or critical-cluster changes, and requires commit kustomize build output in MR so the final manifest diff is visible before merge.
AI-written summary · about
chapters (25)
- 0:00Intro
- 1:12This Session
- 1:59Background: GitOps Managed Desired State
- 3:19Initial Repo Structure - ManyRepo Many Branch
- 4:26Branch Per Cluster/Environment
- 5:09Helm
- 8:03Why We Chose Kustomize
- 11:11How We Manage ArgoCD Configuration
- 13:22Overlay kustomization.yaml
- 13:38What Happened on 13.11.22
- 14:24Change In Overlay argocd-cm
- 14:55Tests Before Merge
- 15:50How We Fixed The Problem
- 16:23The Problem - Expected vs Actual Final argocd-cm Manifest
- 17:05The Problem Impact
- 17:37Overlaying File Contents
- 18:43Patching Multiline Yaml
- 19:28Understanding the Impact of a Change
- 20:32Git Repo Protection
- 23:28MR Pipeline Implementation
- 24:45Human Approval for Critical Cluster Change MR
- 25:10Human Approval for Base Change MR
- 27:06Impact of a Change - Diffing New and Existing
- 27:33Aftermath
- 29:00Summary
related talks
07mLightning Talk: Using Kustomize KRM Functions to Enhance Argo CD Application Deployments
48mHow a Couple of Characters (and GitOps) Brought Down Our Site
14mEase the Pain of Platform Engineers with Argo CD by Leveraging Kustomize Templates
29mHow to Preview and Diff Your Argo CD Deployments
21mSet It To Auto: Using Argo CD Autopilot To Simplify Everything
21mConfiguring Argo CD Is Messy: Here's How We Can Fix It
39mSolving configuration drift using GitOps and ArgoCD
37mEffective Disaster Recovery: The Day We Deleted Production