Using MongoDB with Kafka - Use Cases and Best Practices
summary
Covers concrete patterns: MongoDB as Sink (consume Kafka topics into collections) and as Source (change streams → Kafka), uses spring kafka in TransferWise’s activity service and a custom connector that works in production, compares Debezium-style and MongoDB Kafka Connector features (snapshot mode, parallel tasks, dead‑letter queues), recommends schema registry/Avro for compact serialisation, shows throttling via Kafka consumer/producer rate limits tied to Prometheus alerts and replaying Kafka as a journal for partial recovery.
AI-written summary · about
chapters (17)
- 0:00Intro
- 1:34Agenda
- 2:06What is MongoDB?
- 2:47What is Apache Kafka?
- 3:59Use cases - Topologies
- 4:51Selective Replication/EL/L MongoDB doesn't support selective Replication Oplog or Change Streams (prefered method)
- 6:56Debezium
- 8:38MongoDB Kafka Source Connector
- 10:24Throttling
- 15:00Workload isolation
- 16:49Continuous aggregations Useful for use-cases that raw data are useless for not very useful Kafka streams is your friend - Windowing
- 17:48Journal
- 19:43TransferWise: Activity Service
- 21:33spring-kafka
- 24:35MongoDB Kafka Sink connector: Configuration
- 28:10Recap/Takeaways There are tons of use-cases for MongoDB & Kafka
- 28:58Thank you!!!
related talks
46mStreaming ETL on the Shoulders of Giants
39mKafka as a Platform: The Ecosystem from the Ground Up
41mKafka as a Platform: The Ecosystem from the Ground Up
42mData Streaming for Microservices using Debezium
02h 05mA Deep Dive into Apache Kafka This is Event Streaming
01h 00mKafka as a Platform: the Ecosystem from the Ground Up
01h 00mKafka as a Platform: the Ecosystem from the Ground Up
41mChange Data Streaming Patterns in Distributed Systems