Streaming Platforms in Event-Driven Systems: Patterns, Pitfalls, and Performance
summary
Architecture shows a concrete pipeline: DynamoDB CDC streams feed Kafka (MSK), Apache Flink reconstructs logical ordering and updates Aurora, and an alternative Lambda path lowers latency but loses cross-partition ordering. Key claims: Apache Kafka provides idempotent producers and transactions to approach exactly-once semantics, ordering is only guaranteed per partition, and preserving order reduces throughput. Target metrics are under 100 ms latency with 50–100 consumers. Operational complexity, storage costs, and partition-key design are the main pitfalls.
AI-written summary · about
chapters (20)
- 0:00Introduction and audience setup
- 1:07From monoliths to event-driven architectures
- 3:36Core concepts of event-driven systems
- 7:09Characteristics of events in EDA
- 9:19Benefits of event-driven architecture
- 12:47Challenges in event-driven systems
- 15:16Synchronous vs asynchronous communication
- 18:05Understanding coupling in architectures
- 19:51Introduction to event streaming platforms
- 20:12Why use event streaming systems
- 21:16Fundamentals of streaming platforms
- 24:30Ordering and delivery semantics
- 28:24Delivery semantics explained
- 34:26Scaling streaming platforms
- 36:56Managed Kafka cluster options
- 38:44Common event-driven patterns
- 41:13Real-world e-commerce example
- 42:58Implementing CQRS with Kafka
- 48:38Key takeaways and considerations
- 50:44Q&A and closing remarks
related talks
36mStream All The Things!
30mEvent Streaming and the Future of Applications
23mEvent-Driven Architecture: Unleashing responsiveness
47mStream All the Things!
42mIt's like 10,000 streams when what you need is a queue
29mData Mesh Streaming Patterns: A Real-time Perspective
27mOptimize Your API with Event-Driven Microservice Architecture.
59mBuilding Data Intensive Microservices with Apache Kafka