Six Little Lines of Fail
summary
Six innocuous lines post to Stripe, SendGrid and RabbitMQ inside one DB transaction and produce three real failures: missing emails, downstream systems not updated, and customers charged despite an error page. Mitigations explored include Stripe idempotency and authorization & capture, retry vs compensating refunds, storing outbound messages in an outbox pattern, and moving work to a background process manager. Implementation chosen first is try/catch plus refund, later refactor to async backend workflow; result: safer, slower, more reliable.
AI-written summary · about
chapters (25)
- 0:00Intro
- 1:026 little, innocuous lines of code
- 3:08Failure is ALWAYS an option
- 11:38We can't put the world into a transaction
- 12:33Assume failures
- 12:43Plan for the worst
- 13:204 options for dealing with failures between two systems
- 18:57The Master Plan
- 19:44For each external resource, examine possibilities and pick best option based on business requirements
- 19:58First up: Stripe
- 21:16Retry
- 25:30Undo
- 28:50Option 4: Coordinate
- 32:27Next up: SendGrid
- 35:19Apologize?
- 35:47Finally: RabbitMQ
- 39:54Compensating message is... weird
- 41:25Taking a step back
- 41:56Can we avoid exceptions by breaking the process coupling?
- 50:01Still not there...
- 53:02Starting the workflow
- 55:03Routing slip
- 56:13Saga pattern
- 57:32Process manager pattern
- 58:18Orchestration or choreography?
related talks
58mSix Little Lines of Fail
31mThe Anatomy of a Cascading Failure
29mServices, Scale, Backgrounding and WTF is going on here?!???
01h 01mMessaging: The fine line between awesome and awful
27mReliable Service Integration
30mUnhappy Path & Dealing with Bad Events
38mBuilding For Gracious Failure
55mSo You Want to Maintain a Reliable Event Driven System