Realtime bot detection with Flink
summary
Yelp builds a Flink-backed Bot Signal Platform that processes 2.5 billion events per day and holds about 1 TB state to classify users as bot or not in near real time. It groups activity into session windows, overrides the session trigger with a hard cutoff session trigger to prevent unbounded state, computes per-session features with TTL to carry context across sessions, applies rule and ML models, and publishes classifications to Kafka, a data lake, and a Cassandra-backed API; joins use a latest-classification selection to avoid duplicates.
AI-written summary · about
chapters (6)
- 0:00Intro
- 0:20Yelp's Mission
- 0:54Why bots are a problem?
- 4:15Problem: Ever growing sessions
- 6:27Problem: Inconsistency across sessions
- 10:25Problem: Duplicate Classifications
related talks
29mReal Time Fraud Detection with Stateful Functions
25mReal-Time Fraud Detection at Scale
40mLeveraging Streaming-Based Outlier Detection and SliceLine to Stop Heavily Distributed Bot Attacks
31mKeeping Redditors safe in real-time with Flink Stateful Functions
43mLarge Scale User Behavior Analytics by Flink
01h 00mRock ‘Em Sock ‘Em Robots: Bot Swatting Like the Pros
29mBuilding Resilient AI Platforms: Financial Fraud Detection
16mAnalysing Real time Activity Streams for Security