A Reactive Architecture Approach for a Tax Information System Modernization
summary
Modernizes a provincial tax system into a reactive, event‑driven pipeline using CDC/outbox → Apache NiFi ETL → Kafka (≈90 topics/partitions) → Akka Streams + Kafka workers persisting with event sourcing and CQRS into Cassandra, deployed on Kubernetes. Handles an initial 100 million records weekend initialization and 2–8M events/day ongoing; fixes a throughput bug where Kafka transactions prove non partition‑aware by switching to a committable-partition source and adding idempotency; uses snapshots for recovery, Zipkin/Kafka UI and custom metrics for end‑to‑end observability.
AI-written summary · about
chapters (14)
- 0:00Copernicus project intro
- 1:06Legacy system challenges
- 3:16Non-functional requirements
- 4:43Reactive architecture overview
- 8:58Business process modeling
- 11:33Project execution challenges
- 12:38Improving messaging throughput
- 18:45Kafka consumer config
- 21:56Consumer pipeline code walk
- 22:19Observability in distributed systems
- 26:11Schema evolution challenges
- 30:08Event sourcing schema versions
- 30:49Project timeline rollout
- 32:16Key takeaways
related talks
37mReactive Architecture on AWS: from 0 to 1.6M events/minute
49mRethinking Reactive Architectures
47mThe Rise of Reactive Microservices
54mReacting to an Event-Driven World
51mExperiences from large project maintenance at the Norwegian tax authority
53mBuilding Event Driven Services with Apache Kafka and Kafka Streams
40mDecomposing a live application with confidence
43mImplementing event-sourced microservices