Machine Learning and Cyber Security - Detecting malicious URLs in the haystack
summary
Claims a workable ML pipeline that flags malicious URLs using open datasets and common tooling: training on Alexa top 1M as benign and PhishTank as malicious, extracting WHOIS dates, host IP geolocation, lexical features, digit/special‑char ratios and embeddings, and finding Shannon entropy unhelpful. Models include random forest, character embeddings and deep nets trained with fastai / PyTorch / TensorFlow; best model reaches roughly 90% accuracy on held‑out tests but shows nontrivial false negatives; deployment stack is under construction (Starlette/Flask).
AI-written summary · about
chapters (17)
- 0:00Intro
- 1:38A little story...
- 3:03How our project started
- 3:31Design Thinking
- 5:32Know The User - Challenges
- 6:15Know The User - Security Analyst Persona
- 7:00Nail the problem
- 8:40Ideate - One Prototype At A Time
- 11:02Data Acquisition & Understanding
- 11:59Feature Engineering
- 12:54Modeling & Evaluation - Cross Validation
- 13:58Modeling & Evaluation - Candidate Models
- 16:38Modeling & Evaluation: F1 Score
- 17:50Modeling & Evaluation: Confusion Matrix
- 18:04Modeling & Evaluation: Explainability
- 19:36Modeling & Evaluation - Parameter Tuning
- 20:47Deployment
related talks
32mAI/Machine Learning: What is Actually Working in Cybersecurity?
51mDeepPhish: Simulating Malicious AI
05m[Flash talk] MUDPIPE - Malicious URL Detection for Phishing Identification and Prevention
51mDeepPhish: Simulating Malicious AI
48mCatch me if you can: Building a Web Malware Analyzer using Machine Learning
31mAI-Powered Solutions for Cybersecurity
39mDetecting Website Intrusion and Account Compromise with Machine Learning
31mBuild threat detection models using ML