PIPEFAIL: How a Missing Shell Option Slowed Cloudflare Down
summary
A bash cronjob running in Kubernetes silently returns an empty TOML when an addressing API query slows, because the script lacks strict error handling (missing shell option) and writes that empty value into Quicksilver. The edge daemon dosd reads the empty field, crash loops, and Frontline experiences 5-second head-of-line blocking, causing ~45% CPU drop and ~20% traffic loss for about 30 minutes. The failure needs the rollback timing at :10 and the Thundering Herd against the addressing API to coincide.
AI-written summary · about
chapters (10)
- 0:00Pipefail and Cloudflare intro
- 2:33Key concepts and architecture
- 4:58Change that triggered incident
- 6:01Impact on system metrics
- 7:03Initial remediation steps
- 8:28Root cause investigation
- 10:37Why Frontline slowed down
- 11:40Timing's role in failure
- 13:50Lessons learned
- 16:57Summary
related talks
37mStop Hitting Yourself!
37m10 Ways to Shoot Yourself in the Foot with Kubernetes, #9 Will Surprise You
29mKubernetes Exposed: One Yaml Away from Disaster
27mWhat Doesn’t Kill You Makes You Stronger: The Vulnerabilities Th...
29mPost Mortems for 4 Years of Remote Execution
25mWe're Still Down: A Metastable Failure Tale
35m10 Weird Ways to Blow Up Your Kubernetes
43mDon’t fall asleep at the wheel … in Cloud