GitOps or a Journey to a Production Incident and Swiftly Back Again
summary
GitOps deployment causes a production DNS outage when a one-line config change makes monitoring agents scrape everything, crashing CoreDNS across a 150-node cluster; 30,000 containers (7,000 in prod) amplify the impact. Investigation uses Kubernetes audit logs and git: 58,000 audit log lines → filtered to 10 relevant entries, commit timestamps point to a single merge that bundled preprod and prod, and a quick revert restores service in five minutes. Fixes: one-change-per-environment, synchronization windows and four-eyes approvals, and a GitOps bot that diffs expanded manifests before merge.
AI-written summary · about
chapters (6)
- 0:00GitOps introduction
- 6:39GitOps extends Kubernetes
- 12:41Production DNS incident starts
- 15:51Root cause investigation
- 20:06Five key learnings
- 27:46Summary and Q&A
related talks
35mGitOps - Operations by Pull Request
47mGitOps - Using Git as your source of truth for build, deploy and observability
29mPutting the Ops in GitOps - New Challenges in Managing Cloud Native Environments
36mGitOps: The Single source of truth
39mObservability to Resolution: The Journey Through a Production K8s Incident
30mUsing GitOps for Kubernetes Reliability at Scale
06mGitOps - At a Glance
40mHow GitOps Changes Everything in CI/CD