An Architecture to Tweet Them All

on YouTube ↗28m

" summary

Claims a production-tested pipeline that ingests Twitter via Spark Streaming (Scala), stores raw JSON in CouchDB, uses Kafka/RabbitMQ for IDs between stages, runs Python analyzers for translation sentiment classification and influence, consolidates per-tweet state machine in the DB so failed steps are retryable via a garbage-collector, and indexes results in Elasticsearch for Kibana dashboards. It handles the free API volume (~2,000 tweets/day), favors CouchDB over HDFS/Hive, and reports the architecture as working and scalable.

AI-written summary · about

" chapters (17)

1. 0:00Introduction
2. 1:08Project goals
3. 2:37Problem solution approach
4. 3:17Technology stack
5. 4:44Database choice
6. 5:30Message queuing design
7. 6:16Collector with Spark Streaming
8. 7:19Analyzer with Python
9. 8:03Actor roles
10. 9:08Failure handling
11. 10:36State machine workflow
12. 13:29Tweet processing example
13. 14:58Visualization with Grafana
14. 16:26Architecture conclusions
15. 17:34Extending beyond Twitter
16. 19:04Improvements
17. 20:47Q&A

" related talks

• 01h 01mStream Data Processing for Fun and ProfitDavid Ostrovsky · NDC Oslo 2017
• 36mData pipelines engineering made simple with ScalaRaphael Claude · Scala.IO 2024 - Nantes
• 30mThe Distributed Database Behind TwitterMehrdad Nurolahzade · Distributed SQL Summit 2020
• 34mBig and Small Data at @TwitterJeremy Cole · O'Reilly MySQL Conference & Expo 2011
• 34mChasing the Stream Processing UtopiaKartik Paramasivam · Strange Loop 2018
• 01h 22mWhen Rotten Tomatoes Isn’t Enough: Analyzing Twitter Movie ReviewsAmanda Moran · PyData Los Angeles 2018
• 35mStorage, Then and NowEvan Weaver, Boaz Avital · Scale By the Bay 2017
• 41mInside Twitter's Realtime Delivery of Tweets for Enterprise CustomersLisa White · Strange Loop 2017