Large Scale Social Network Data Collection & Analysis...
summary
Pipeline description: People Pattern normalizes multi-network JSON into ActivityStreams using Apache Streams (incubating), runs distributed ingestion and processing on Apache Flink, and ingests about 60 million tweets a day and roughly 10 million profiles daily to grow a production graph of ~300 million profiles. Data lands in Parquet, feeds map-only ML jobs that predict demographics, interests and sentiment, then indexes to Elasticsearch, Neo4j and Cassandra and supports on‑demand Flink/Spark reports and Twitter Ads activation.
AI-written summary · about
chapters (21)
- 0:00Introduction and welcome
- 1:04Problem of data silos
- 2:47Activity Streams standard
- 5:38Interoperability challenges
- 7:25Apache Streams project
- 8:50Apache Streams architecture
- 10:13System architecture overview
- 12:00Data collection process
- 13:27Analysis pipeline details
- 14:52Aggregation and indexing
- 16:39Database access patterns
- 17:42API endpoints and features
- 19:06Live demo of application
- 24:47Interest analysis features
- 25:50Reporting framework
- 27:13Activation and targeting
- 28:15Conclusion and Q&A introduction
- 29:08LinkedIn data access discussion
- 31:20Data ownership and GDPR
- 33:14Spark vs Flink comparison
- 35:20Apache Streams examples
related talks
56mSocial Listening, Data Extraction & Discovery
42mBuilding a near real time personalized feed using data streaming and Aerospike
56mReal Time Data Pipelines
01h 05mSocial Text, Sentiment, and Tone Analysis
40mArchitectures for Exploring Your Personal Data
55mSocial Listening for Richer Data, Unstructured Analytics
31mReal time association mining in large social networks
36mData pipelines engineering made simple with Scala