Six Little Lines of Fail
summary
Six tiny lines call a DB, create an order, hit Stripe, call SendGrid, publish to RabbitMQ, then commit a transaction and failure modes make charging without an order and missing emails inevitable. Concrete fixes include Stripe idempotency key or authorize/capture, delaying emails and messages with an outbox pattern inside the DB, and coordinating retries and compensations with a central process manager (saga) plus background workers. The refactor reduces dependencies during checkout and prevents the inconsistent states that caused real customer charges and errors.
AI-written summary · about
chapters (23)
- 0:00Intro
- 1:296 little, innocuous lines of code
- 3:05Failure is ALWAYS an option
- 8:30is probably not good for business
- 9:19into a transaction
- 9:35Assume failures
- 10:034 options for dealing with failures between two systems
- 13:29The Master Plan
- 14:07First up: Stripe
- 14:22Probably don't want to ignore a failed payment
- 16:36Undo
- 20:30Next up: SendGrid
- 21:40Retry
- 24:37Apologize?
- 24:56Coordinate
- 27:28Can't un-send a message
- 28:48Taking a step back
- 34:34Way too much going on in a single button click
- 34:49Modern ecommerce uses background processing to fulfill orders
- 40:19Routing slip
- 41:49Saga pattern
- 43:38Process manager pattern
- 44:17Orchestration or choreography?
related talks
47mFixing Distributed Systems Fail
23mDistributed Systems: Your Only Guarantee Is Inconsistency
42mHow to stay alive even when others go down:...
27mPlatform Engineering for Subscription Payment Systems
42mFive Common Mistakes With Distributed Systems
29mMaking Sense of Chaos: Operational Visibility in Distributed Systems
47mTop 10 Event-Driven Architecture Pitfalls
30mWhat Breaks Our Systems: A Taxonomy of Black Swans