Apache Kafka as a Monitoring Data Pipeline
summary
Shows a working pipeline built with Strimzi on Kubernetes: a 7‑node Kafka cluster receives logs from Fluent Bit → Kafka → Elasticsearch/Kibana, stored in a topic created via the operator. Kafka Connect runs connectors that write batched files to S3 (files closed after 1,000 messages or 5s; example file ~680 KB) and post alerts to Slack. A small Kafka Streams app detects authorization errors, writes alert messages to a topic, and the Slack connector delivers them — the demo proves the flow end‑to‑end.
AI-written summary · about
chapters (6)
- 0:00Introduction to pipelines
- 4:19Kafka architectural benefits
- 10:24Demo: Basic logging setup
- 20:48Demo: Data archiving
- 26:47Demo: Alerting systems
- 31:20Considerations and conclusion
related talks
19mStrimzi & OpenTelemetry: Building End-to-End Observability for Kafka on Kubernetes
02h 05mA Deep Dive into Apache Kafka This is Event Streaming
49mLet’s Build a Streaming Data Pipeline!
41mApache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
29mApache Kafka and ksqlDB in Action : Let’s Build a Streaming Data Pipeline
01h 03mApache Kafka Event streaming platform for .NET developers
29mKafka on Kubernetes: Does it really have to be the hard way?
01h 00mKafka as a Platform: the Ecosystem from the Ground Up