Modern Event-driven Architecture: Adopting Data in Motion
summary
Argues that moving from static databases to event streams changes architecture: Kafka + Confluent become the durable backbone, enabling consumer offsets, replay, and tens of gigabytes a second ingestion; demonstrates ksqlDB for stream processing and Kafka schema registry for backward/forward compatibility and discoverability; shows connectors to bridge legacy DBs and Redshift so BI can query real time views; gives a neobank/credit-bureau example, explains Kappa vs Lambda, tiered storage, and operational patterns for governing, evolving and recovering event-driven systems.
AI-written summary · about
chapters (20)
- 0:00Intro
- 2:14Data is the new engine of business success
- 3:19Paradigm for data in motion: Event streams
- 4:54Basic event oriented
- 6:15Event-command oriented
- 7:47Event oriented - asynchronous
- 9:56Evnet oriented - decoupled
- 11:04Modern architecture
- 12:37The rise of event streaming
- 13:50Anatomy of a Kafka topic
- 16:02Principles of good EDA using data in motion
- 16:391. Treat event data as first class citizen
- 21:142. Make schemas the contract for event streams
- 24:283. Plan for breaking changes and failures
- 26:20Demo
- 28:394. Create & empower a governing body
- 31:195. Provide self-service functionality with first class support
- 32:016. Make data easy to discover & use
- 36:367. Rely on connectors to bridge streams & batch
- 38:35Outro
related talks
40mThe Changing Face Of ETL: Event-driven Architectures for Data Engineers
17mOvercome By Events - Apache Kafka And Event-oriented Architecture
43mBack to fundamentals: from databases to event-driven platforms
41mApache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
33mKafka streams at scale
52mEvent Sourcing for the Cloud Developer
49mLet’s Build a Streaming Data Pipeline!
32mTake back control of your data with Event Driven Systems