Machine Learning on big data in security applications
summary
Avast says it runs Python with Apache Spark and Kafka over HDFS to process hundreds of terabytes and ~450 million samples, extracting 500+ static features, clustering (k‑means) then training per‑cluster random forests. A Spark streaming pipeline builds thousands of per‑detection time‑series anomaly models, deployed in silent mode for validation. K‑means on Spark is slower and produced huge clusters, but Spark eases experiments, parallel training, windowing and checkpointing while exposing stability and scaling challenges.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:14Avast mission and scale
- 3:21Malware classification
- 5:51Spark for pipeline
- 7:40Spark k-means challenges
- 10:17Malware time series anomalies
- 14:44Prototype to production
- 18:39Spark anomaly advantages
- 21:59Key takeaways
- 22:49Q&A
