Workshop: Log Processing with Apache NiFi and Apache Kafka
summary
Hands-on workshop shows NiFi ingesting the NASA Kennedy Space Center Logs (≈3.5 million requests), splitting the ASCII file into one-log-per-line with SplitText, publishing to Kafka topics and consuming with PublishKafka/ConsumeKafka, using UpdateAttribute, ExecuteScript to normalize timestamps to UTC, and RouteOnAttribute to mark IP vs hostname. Data lands into OpenSearch with an index template and ingest pipeline, dashboards render counts and trends; pipeline works end-to-end but hits NiFi flowfile and batch throughput limits.
AI-written summary · about
chapters (8)
- 0:00Introduction and overview
- 1:23Log analytics fundamentals
- 2:10Understanding the tools
- 5:30Data set demonstration
- 7:17Architecture and setup
- 9:15NiFi and Kafka demonstration
- 17:12Data transformation pipeline
- 33:11OpenSearch integration
related talks
35mStream Processing for Analysts with Flink and Nussknacker
43mTurn your logs into actionable data at any scale with AWS
16mLogging Apache Spark - How we made it easy
18mDistributed OpenSearch Monitoring at Scale With Apache NiFi and MiNiFi Agents
48mLogging and ranting
31mThe Long and Winding Road to Application Logging [A]
31mLogging at Scale, Billion+ messages, 100K msg/sec
46mBuilding a scalable logging platform