Event streaming with modern data pipelines in a SaaS architecture
summary
A SaaS company replaces a nightly 20–24 hour snapshot pipeline (slipping to every 2–3 days) with change-data-capture into S3 using AWS DMS, producing near‑real‑time events with typical lag of 30 seconds to a few minutes. Full initial full loads for a few terabytes finish in about 6–8 hours. DMS handles schema/table filtering, native anonymization (with edge cases sent to Glue/Lambda), and fans out via SNS; deployed against three relational databases so far, with DDL pauses scripted to avoid replication failures.
AI-written summary · about
chapters (16)
- 0:00Introduction and welcome
- 1:49About KnowBe4 and Project Unicorn
- 3:57Old data pipeline challenges
- 6:10Evaluating AWS streaming options
- 9:02Why choose Database Migration Service
- 11:50Implementation and architecture
- 13:40Real-world use cases and growth
- 15:12Adding new consumers and producers
- 16:12Latency and performance insights
- 18:00Cost and technical debt benefits
- 22:42Kafka vs Kinesis Streams
- 24:30Latency sources and troubleshooting
- 25:11Aurora cluster configurations
- 26:42Performance impact of DMS
- 27:43Resilience in consumer processes
- 28:27Closing and resources
