There's No Place Like Production
summary
A one-line Helm values overwrite clobbers a nested production-only block and causes a SEV-1 outage that passes CI, staging and acceptance but breaks a security sidecar only present in prod, rollback restores service, and manual diffing of rendered Kubernetes manifests reveals the bug after noisy unstructured ELK logs and no traces. The deployment uses Helm templating plus Argo app-of-apps hierarchy, observability lacks distributed tracing compare traces, incident report takes ~9 hours, and recommended fixes include Helm test and storing rendered manifests and adding tracing and clearer env-specific overlays.
AI-written summary · about
chapters (16)
- 0:00Welcome and introduction
- 0:21Testing in production debate
- 2:31Context of the outage
- 5:00Traffic service overview
- 6:49Planning the one-line change
- 8:53Production deployment and incident
- 10:43Incident response and rollback
- 11:48Cultural reflections on incidents
- 14:17Incident review and learning
- 17:27Root cause analysis
- 19:59Jenga Tower analogy
- 21:04Turning retro into classroom
- 23:14Breaking information silos
- 24:39Production vs other environments
- 25:21Learning culture takeaways
- 26:26Conclusion and Q&A
related talks
09mThe thin line between code and configuration (YAML/Helm rant)
26mHelm your way with Kubernetes
27mAre you Helming it the right way? Helm challenges and its solutions
27mDay 2 Operations of Helm Charts
34mKustomize vs Helm - Civil War
10mUsing Helm to the Fullest
48mHow a Couple of Characters (and GitOps) Brought Down Our Site
41mHelm - the Better Way to Deploy on Kubernetes