Services, Scale, Backgrounding and WTF is going on here?!???
summary
Extraction from a monolith breaks synchronous assumptions: queuing via Resque + Redis can enqueue an ID before the DB transaction commits, producing unusable accounts under a unique-email constraint. Fixes applied include wrapping create+enqueue in DB transactions, retrying the job a few times with backoff (usually succeeds within five attempts), and moving side effects out of callbacks or to after_create vs after_commit where available. Long-term fixes: require idempotent request ID for mail, structured logging, DB constraints, audits and periodic sanity checks.
AI-written summary · about
chapters (18)
- 0:00Intro
- 0:36Services, Scale
- 2:25Problem
- 5:02Timeout to Redis
- 6:20Solution
- 10:24Better(ish)
- 14:30Reasonable?
- 15:02Mailers
- 17:35An event fails!
- 20:05Make Idempotent
- 20:50Additional calls
- 21:31What if...
- 22:33Historical Record
- 23:54Prevent Bad Data
- 25:25Fix Errors
- 26:16Extract Services
- 27:18In Review
- 27:49Thank You! @davetron 5000
related talks
40mUncoupling Systems
01h 02mSix Little Lines of Fail
43mSleeping on the job
34mOh S***: How to bring a big Rails website down (and how not to)
40mFacing the Monolith Overcoming Monolithic Applications with SOA
58mSix Little Lines of Fail
30mUnhappy Path & Dealing with Bad Events
31mThe Anatomy of a Cascading Failure