Leveraging Serverless architecture with Confluent Kafka
summary
A real system is built: AWS Lambda producers and consumers wired to Confluent Kafka topics and partitions, backed by DynamoDB and SQS dead‑letter queues, achieving parallel notification delivery for earthquake data from the national institute. The project claims one to three minutes of warning, uses custom DynamoDB locking and segmented worker partitioning to avoid duplicate sends, and autoscaling via Terraform/GitHub Actions adjusts capacity for hundreds to a few thousand subscribers.
AI-written summary · about
chapters (9)
- 0:00Introduction and Session Overview
- 2:24Project Background and Motivation
- 2:42Current Earthquake Alert Systems
- 4:41Data Sources and Initial Challenges
- 7:58Serverless Architecture and Design
- 9:24Detailed System Architecture
- 15:58Kafka Integration and Optimization
- 28:54Auto Scaling and Future Improvements
- 34:05Conclusion and Future Directions
related talks
40mFrom Tremors to Saving Lives: Serverless Kafka Arch for Earthquake Notifications
40mFrom Tremors to Saving Lives: Serverless Kafka Arch for Earthquake Notifications
12mServerless Stream Processing with Confluent Cloud, Kafka and AWS Lambda
28mBuilding Event Streaming Applications with Serverless Kafka
42mBuilding real-time serverless data applications
47mServerless Kafka Patterns.
32mBuilding and scaling a log analytics platform - a serverless approach
47mEvent-driven Serverless Architecture with DynamoDB