Alice and Mad Hatter....
summary
Built a Scala Spark pipeline that turns raw records into cached RDD vectors, applies RDD vector normalization (mean/stddev), trains KMeans (rerunning to avoid order sensitivity) as a KMeans single-cluster anomaly detector, computes distances to the centroid with a tunable threshold, and emits anomaly scores. Execution targets Spark on Kubernetes via spark-submit in cluster mode using a service account and container image. Pipeline produces predictions and flagged anomalies; story ends with the investigation continuing.
AI-written summary · about
chapters (18)
- 0:00Introduction and speakers
- 1:19Alice and Mad Hatter adventure
- 3:02Recap of Alice’s previous journeys
- 6:15Meeting the Mad Hatter and Caterpillar
- 8:03Solving the riddle with anomalies
- 11:37Machine learning pipeline overview
- 15:07Collaboration between data engineers and scientists
- 16:34Choosing Scala and Spark for the solution
- 17:16Data ingestion and preprocessing
- 18:01Normalizing data for model accuracy
- 19:25Building the anomaly detection pipeline
- 21:12Training the K-means clustering model
- 22:36Running Spark on Kubernetes
- 24:22Spark application configuration on Kubernetes
- 26:29Interpreting binary data hints
- 27:13Realizing it was a dream
- 28:17The past is the key to the future
- 28:59Q&A and closing remarks
related talks
19mBig Data
51mApache Spark and machine learning: dreams and realities
30mDeploy end to end ML pipeline using Apache spark ...
27mMachine Learning on big data in security applications
52mDevil in the Haystack
50mApache Spark vs Machine Learning
39mMachine Learning with Scala on Spark
27mBeyond Logs: Real Time Anomaly Detection with Machine Learning