Tyranny of the Clock
summary
A GitLab.com incident response story tracing intermittent git-over-SSH failures to early TCP terminations caused by synchronized cron spikes. Packet captures and HAProxy + BigQuery logs reveal connections always die after exactly eight packets and bytes read 0; OpenSSH’s MaxStartups default was limiting at 100 then raised to 250–300, reducing errors from ~1.5% to ~0.001%. Deploying HAProxy’s rate-limit and switching to leastconn backend balancing evens load and eliminates the spikes; metrics and alerting rules are rewritten to catch recurrence.
AI-written summary · about
chapters (21)
- 0:00System Administration
- 4:10Packet captures
- 4:56Lesson #1
- 5:39Conversations
- 6:15Logs logs and more logs - BigQuery to the rescue
- 7:37Tom down connections
- 10:24Lesson #2
- 10:42SSH Config
- 12:41Bump it
- 13:23How high could we go?
- 18:00Lesson #4: leave crumbs
- 19:09A delightful graph
- 19:27Lesson W5
- 20:10A bit of ret-con math
- 20:37Error rates
- 21:41Lesson W6
- 22:47Followup: Why u no alert?
- 24:31Detecting using mtail
- 26:26Visualize the Tyranny
- 28:01Make people like me happy or sad with your scheduling
- 28:46Final thoughts
related talks
37mStop Hitting Yourself!
28mMy Favourite Errors
29mPost Mortems for 4 Years of Remote Execution
29mBest Practices for Kubernetes Runners
21mWhen the Cloud Fails: Debugging the "Undocumented"
42mHow to stay alive even when others go down:...
22mOops, I deployed too hard
35mGitOps or a Journey to a Production Incident and Swiftly Back Again