Hunting Criminals with Hybrid Analytics
summary
A practical system detects insider trading by combining separate models for NLP, graph linkage and time-series routines into an ensemble that retrains in production from human feedback. Data flows through Spark Streaming + Kafka at terabytes per day, features use scikit-learn, nltk, statsmodels inside Jupyter notebooks and packaged PySpark UDFs, and the pipeline ranks suspects because true alerts are about one in a million. The open-source notebooks and AWS jumpstart reproduce feature engineering, online retraining, and investigator feedback loops.
AI-written summary · about
chapters (11)
- 0:00Introduction and speaker background
- 0:54Problem of rogue traders in banking
- 2:39Challenges in detecting suspicious behavior
- 5:48Hybrid analytics approach explained
- 9:15Key algorithms and models needed
- 11:03Open-source demo and reference architecture
- 15:57User-level feature engineering
- 18:47System learning in production
- 20:12Q&A on rare event detection
- 21:14Testing classifiers in production
- 23:27Summary and closing
related talks
52mDevil in the Haystack
22mDavid Talby, SVP Engineering, Atigeo
25mReal-Time Fraud Detection at Scale
48mHunting Stealth Adversaries with Graphs & AI
31mFinding Needles in a Growing Haystack
27mMachine Learning on big data in security applications
45mThreat Hunting with Python and Pandas
20mFighting Corruption with Technology