Sentiment Analysis in Action: Building Your Real-time Pipeline
summary
Real-time pipeline ingests social media records into Apache Kafka, processes them with Apache Flink calling AI models, then indexes results into OpenSearch for live aggregation. Two model approaches run: calls to the GPT API for multi-field JSON replies (language sarcasm category sentiment) and lighter classifiers (distilBERT/Twitter‑RoBERTa) for probabilistic sentiment. Demo shows working end‑to‑end but exposes issues: inconsistent label formats, occasional JSON parsing failures, and edge cases like sarcasm and negation that reduce accuracy.
AI-written summary · about
chapters (10)
- 0:00Real-time sentiment analysis intro
- 2:16What is sentiment analysis
- 7:58Streaming with Kafka
- 10:31Why Flink for streams
- 15:52First ChatGPT API attempt
- 23:19JSON prompts for better responses
- 25:30Alternative sentiment models
- 32:20BERT versus generative models
- 36:39Starting with sentiment models
- 39:10Q&A
related talks
47mAI vs. Karen: Real-Time Customer Sentiment Detection at Scale
01h 06mAI vs. Karen: Real-Time Customer Sentiment Detection at Scale
01h 05mSocial Text, Sentiment, and Tone Analysis
41mAdding Generative AI to Real Time Streaming Pipelines
01h 47mRadanalytics.io streaming applications workshop
39mSentiment Analysis, its techniques and applications
21mLabels to Inference: A Continuous Sentiment Pipeline
01h 25mBuild and Deploy an End-to-end Streaming NLP Insight System