Where we're going... we don't need batch jobs
summary
Argues that recurring batch jobs become unnecessary by modelling future events as persistent timeouts: implement a saga that maintains a single-row sliding week-total and sends a delayed message to subtract each order one week later. Uses durable queuing (Azure Service Bus / SQS tricks, RabbitMQ TTL) plus an outbox pattern to get atomicity. Describes a binary-delay queue technique (28 queues → ~8.5 years max delay), proven workable and testable in production.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:44Runtime behavior problems
- 5:32Batch jobs fail for rules
- 10:23Replace batch with sagas
- 22:14Sagas in NServiceBus
- 30:01Saga transactionality with outbox
- 35:36Batch jobs wrong for reporting
- 41:12Learning sagas and stateful handling
- 42:57Testing sagas with TDD
- 47:31Organizational challenges
- 51:13Q&A
related talks
58mWelcome to the (state) machine
39mDealing with eventual consistency
51mDealing with eventual consistency
01h 01mMessaging: The fine line between awesome and awful
44mThinking Asynchronously: App Integration Patterns for Microservices
59mSucceeding at Reactive Architecture
46mShifting Gears: From Events to Event-Driven
31mWriting Reliable, Scalable, Fault Oblivious Code On K8s the Easy Way