Intelligent Log Analysis and Real-time Anomaly Detection @ Salesforce
summary
Salesforce processes tens of gigabytes per second firehose and runs a public-cloud, elastic multi-tenant platform that performs real-time parsing, joins and anomaly detection using Apache Flink windowed keyed state. Models are mostly unsupervised, millions of tenant-specific models are trained or registered (offline on Kubeflow/TensorFlow and online in Flink), inference uses async RPCs, joins tolerate partial results, and per-tenant scaling uses per-tenant Application Master sessions on YARN/Kubernetes/Mesos serving hundreds of tenants and users.
AI-written summary · about
chapters (7)
- 0:00Intro
- 2:19Challenges
- 15:45Multitenant
- 16:55Windows Statistics
- 19:08ML
- 20:52Dashboards
- 25:20Biggest challenges
related talks
16mAnalysing Real time Activity Streams for Security
43mBeyond Logs: Real-Time Anomaly Detection with Machine Learning
29mTracing Service Dependencies at Salesforce
49mImpact on business and society
38mReal Time ML Pipelines in Multi-Tenant Environments
27mDataEngBytes 2023 - MEL-T1-08 - Troy Sellers
27mBeyond Logs: Real Time Anomaly Detection with Machine Learning
25mReal-Time Fraud Detection at Scale