Kafka on the Fly: A Serverless Approach to Data Streaming
summary
Serverless Kafka using Upstash and KafkaJS demonstrates a concrete pipeline: a Docker Compose microservice produces to a topic and a consumer stores JSON user records into MongoDB, with HTTP endpoints on 8080/8081. The claim is that managed, HTTP-accessible Kafka removes cluster ops and auto‑scales; the demo shows messages sent from Postman are received and persisted. Practical limits shown include free‑plan retention ~256 MB and 1 MB max message size.
AI-written summary · about
chapters (13)
- 0:00Introduction and speaker intro
- 1:48Why real-time data streaming
- 6:11Apache Kafka overview
- 9:44Kafka core concepts
- 13:59Kafka architecture flow
- 14:21Serverless technology explained
- 17:30Why use serverless
- 17:52Challenges with traditional Kafka
- 20:19Introducing Upstash Kafka
- 26:41Upstash Kafka benefits
- 28:11Live demo setup
- 34:20Demo execution and results
- 36:05Key takeaways and conclusion
related talks
28mBuilding Event Streaming Applications with Serverless Kafka
28mThe Future of Streaming: Global Apps, Event Stores and Serverless
12mServerless Stream Processing with Confluent Cloud, Kafka and AWS Lambda
47mServerless Kafka Patterns.
36mLeveraging Serverless architecture with Confluent Kafka
54mHomemade Serverless: Making it from Scratch
31mServerless Java in Action: Cloud Agnostic Design Patterns and Tips
46mServerless Deployment of Analytics