Actionable Insights with Real-time Streaming Analytics of Customer Reviews
summary
A working pipeline streams Amazon movie reviews from Beats into Kafka, processes them with Spark Streaming that applies a simple tf-idf classifier trained on ~2,000 labeled reviews, writes predictions back to Kafka and into Elasticsearch via Logstash, and shows live dashboards in Kibana with ~600+ reviews indexed and auto-refresh every 5s. The system runs end to end and meets low-latency throughput goals, but model quality is modest — roughly 400 correct vs 228 mismatches out of ~700 counted — so NLP or embeddings are recommended.
AI-written summary · about
chapters (16)
- 0:00Intro
- 2:23Unstructured Data
- 5:53Blueprint - Reference Architecture
- 8:26Data Transport - Why Beats?
- 9:45Data Integration - Why Kafka?
- 10:41Data Integration - Why Logstash?
- 11:35Data Storage - Why Elasticsearch?
- 13:17Data Processing - Why Spark?
- 14:05Data Processing - Why Zeppelin?
- 15:08Data Visualisation - Why Kibana?
- 16:04Demo Setting - The Data
- 18:12Demo Setting - Workflow
- 26:02Recap - Requirements met?
- 29:05Streaming Analytics Use Cases
- 31:28Possible Future Improvements
- 33:42References
related talks
45mReal-time, Streaming Advanced Analytics....
43mDetecting fake reviews at scale using Apache Spark and John Snow Labs
01h 47mRadanalytics.io streaming applications workshop
01h 22mWhen Rotten Tomatoes Isn’t Enough: Analyzing Twitter Movie Reviews
29mApache Kafka and ksqlDB in Action : Let’s Build a Streaming Data Pipeline
27mBeyond Logs: Real Time Anomaly Detection with Machine Learning
42mHow to Leverage the Apache Kafka Ecosystem to...
01h 02mBig Data Analytics in Near-Real-Time with Apache Kafka Streams