Time-based domain events - how to deal with time
summary
Design uses Kafka Streams with state stores backed by RocksDB to postpone and replay time-based domain events for ~450,000 employees and ~9,000 leavers/month, with day precision. Implementation stores future events keyed by date, schedules via the Streams processor API, tests with TopologyTestDriver, and forwards effective events to an output topic. It works but exposes caveats: lexicographic range queries cause duplicate forwards and require a cleaner, state stores lack TTL, RocksDB increases memory/disk, and wall-clock scheduling is best-effort.
AI-written summary · about
chapters (32)
- 0:00Intro
- 2:35Employee Experience Expertise
- 3:00Rewards And Benefits
- 3:22Employee Work Cycle Context
- 4:27Case 1 Discount Eligibility
- 5:11A niche of Time-based domain events
- 6:53Recap • Expiring time frame
- 7:29Case 2 Just in time events
- 8:41Events for future
- 9:46Retroactive event
- 9:57ACL Anti-corruption layer
- 10:17Event Scheduler
- 10:37Events streaming
- 10:53Drivers
- 13:07Kafka Streams DSL
- 14:01PAPI Kafka Streams Processor API
- 14:22Processor Context
- 14:32Processor Time
- 16:03Where put not effective events?
- 16:20State Stores
- 21:16Scenarios
- 23:00Schedule topology with Cleaner
- 23:26Are we done?
- 23:50The idea
- 24:40Reschedule Topology
- 26:08What next?
- 26:54Final Topology
- 27:08Achievements
- 28:55Considerations
- 31:11Takeaways
- 32:52Alternatives
- 34:47References
related talks
30mBuilding domain specific databases with a distributed commit log
49mEvents... And what next?
58mLet's build the worst Event Sourcing system!
30mEvent Streaming and the Future of Applications
53mBuilding Event Driven Services with Apache Kafka and Kafka Streams
26mTen Best Practices for Event Streaming Success
46mState or events? Which shall I keep?
31mDesigning a mission critical event system