Scaling Security Anomaly Detection in Enterprise Knowledge Base with Dask
summary
The pipeline ingests metadata from SharePoint Confluence and Google Drive, canonicalises it to CSV, applies NMF/SVD and rule‑based plus sensitive‑pattern detection, and emits jsonl security alerts. It runs on terabytes of data—2.5 million documents (~2 TB) with ~5 GB metadata—and initially hits OOMs and task‑graph stalls. Fixes include custom loaders, strategic repartitioning, SSD spill, and persisted intermediate results, debugged with the Dask dashboard. The system successfully surfaces permission mismatches and real sensitive‑data leaks such as exposed credit‑card numbers.
AI-written summary · about
chapters (12)
- 0:00Scaling security anomaly detection
- 2:13Why we built the pipeline
- 6:31What is Dask and why
- 9:04Live Dask demo in JupyterLab
- 14:23First production run failures
- 15:47Dask workers memory and tasks
- 17:16Solving memory and skew issues
- 19:01Avoiding graph overhead
- 21:36Scheduler choke and resources
- 23:22Debugging Dask tools
- 24:48Final results and lessons
- 25:30Q&A
related talks
23mUsing machine learning to detect sensitive documents on SharePoint
46mProtecting sensitive data in huge datasets: Cloud tools you can use
31mLeverage AI to Protect Your Data Assets... from AI
21mSaaS Compliance Automation with a Python Stack
29mThresholds Are for Old Threats: Demystifying AI and Machine Learning to Enhance SOC Detection
31mBuild threat detection models using ML
01h 00mAutomating Security in Your Data Pipeline
24mYour company, as a Knowledge Graph - the foundation of cybersecurity’s future