The Doctor Is In: Using checkups to find bugs in production
summary
Claim: lightweight “checkups” run in production to detect silent failures. Implementation details: small Ruby tasks (rake jobs, after_commit hooks, background jobs) scheduled with Heroku Scheduler, periodic scans calling ActiveRecord#valid, and spot checks of cached balances. Surprising findings: a race condition deletes all emails for a user and a caching provider causes randomly divergent balance results. Outcome: the checkup suite alerts and sometimes auto-mitigates (clears cache), letting teams find and fix bugs before users notice.
AI-written summary · about
chapters (12)
- 0:00Intro
- 0:35Introduction
- 8:14Problems with this approach
- 15:00Writing a checkup
- 16:00Zero email addresses
- 17:57Writing a check up
- 20:06When should I write a check up
- 23:25When should I use a checkup
- 25:50Splitwises checkup suite
- 28:08A critical infrastructure problem
- 30:26Final thoughts
- 32:36My suggestion
related talks
28mDo You Know If Your Service Is Working Properly?...
35mThe Virtuous Cycle: Getting Good Things out of Bad Failures
35mTesting in Production: there is a better way
32mOperating Rails: what about after you deploy?
48mWhen things go wrong
35mSite Reliability on Rails
49mObservability in Practice: How to Become Proactive with Bugs
41mDetective Hat: Investigating Production Issues