When One Line Took Thousands of Websites Offline
summary
A single line change in the GitLab CI deployment disables a CRD version and triggers an auth operator pointed at the staging API to delete resources, wiping 1,200 Drupal sites (part of >8,000 hosted) within 20 minutes. Infrastructure runs on OpenShift OKD4 with custom operators; recovery uses valo to restore custom resources and relies on soft-delete volumes (30‑day detach) while DB point‑in‑time restores by the DB team complete full recovery by day two. Pipeline preview diffs and stricter deployment checks are added afterward.
AI-written summary · about
chapters (6)
- 0:00CERN web infrastructure intro
- 6:39Drupal cluster importance
- 6:59Single line change incident
- 14:05Root cause of outage
- 18:00Lessons learned improvements
- 25:46Small changes and recovery
related talks
29mRobust delivery with CI/CD
21mHandling a Major Outage Transparently
48mHow a Couple of Characters (and GitOps) Brought Down Our Site
35mGitOps or a Journey to a Production Incident and Swiftly Back Again
20mCERN's 1500 Drupal Websites on Kubernetes: Sa...
21mDriving Cultural Change to Improve Software Development and CI/CD
22mOops, I deployed too hard
46mFrom 1,000 to 5,000 Users: Scaling GitLab Without (Too Much) Pain