Fighting phishing and spam with online machine learning on data streams
summary
FreshMail builds an online anti-abuse engine called SendGuard that processes thousands of messages per second and over 70M messages a day, using FlinkML and Flink CEP for stream processing. Pipeline components include Kafka, Hadoop, Spark, Elasticsearch and Kubernetes; models deploy as REST services and use online methods like SGD/partial_fit and TensorFlow checkpoints. Feature engineering yields roughly 150–200 features, rapid event-aligned rules precede ML, and immediate link-blocking and CEP handle shifting phishing patterns.
AI-written summary · about
chapters (8)
- 0:00Introduction and background
- 3:21Spam and phishing challenges
- 7:37Building the data platform
- 10:52Machine learning necessity
- 15:03Online machine learning approach
- 18:18Model deployment strategies
- 20:07Learning from data streams
- 21:56Lessons and future plans
related talks
23mML Platform Challenges & Applications at Freshworks
37mAutomated Machine Learning: Building a Pipeline
25mReal-Time Fraud Detection at Scale
12mWelcome to Spam Fighting @Scale
16mAnalysing Real time Activity Streams for Security
29mGuardians of the Gateway: Battling SMS Fraud in Real Time
24mHighly Scalable Discriminative Spam Filtering
16mProduction MLOps for Spam Detection: Real-Time Multi-Modal AI