How to numerically represent semi-structured log data for...
summary
Pipeline converts raw logs into embeddings and normalization rules and evaluates anomaly detection on real data rather than promises. Uses a character‑ngram embedding (better than word2vec), DBSCAN with cosine distance, and regex rules derived from clusters; trains on 22 million events and shrinks vocabulary about sevenfold to 75,000 unique words with ~30 rules. Builds an auto‑encoder with 3‑layer GRU encoder/decoder, gets baseline reconstruction error ~0.004 and shows higher error for shuffled chains so the method proves workable under production constraints.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 1:11Log data and NLP history
- 3:45Semi-structured log challenges
- 5:55NLP models for anomaly detection
- 7:21One-hot encoding limits
- 10:39Word embeddings for logs
- 13:09Context for log embeddings
- 14:17Training embeddings on logs
- 15:49Normalization rules from clusters
- 17:41Anomalies in process chains
- 19:32Auto-encoder for anomalies
- 20:37Summary and future
- 21:17Q&A
related talks
26mNLP for security log analysis : Learning to crawl before you run
10mUtilizing Word Embeddings and Gradient Boosting
21mAnomaly Detection for Platform Logs using Embedded Context
10mML-based Log Analysis for Faster Debugging
27mLog messages processing using NLP tools
48mHunting Stealth Adversaries with Graphs & AI
17mDetecting anomalous sequences using text processing methods
32mMonitoring in the era of Generative AI, LLVMs, and embeddings