Fast Log Analysis Made Easy by Automatically Parsing Heterogeneous Logs
summary
Automatic parser called LogMine clusters tokenized logs, masks datatypes, aligns variants with Smith-Waterman and outputs Logstash filters that integrate with ELK. It finds a small set of patterns by building a pattern tree rather than relying on hand written regex. Demo processes ~15,000 PDU lines in 9 seconds, and 250,000 messy business-process lines in 27 seconds producing 339 default patterns. Memory and CPU scale with level‑one uniqueness, so millions of lines reduce to few representatives for fast downstream parsing.
AI-written summary · about
chapters (20)
- 0:00Intro
- 2:29Log Parsing: Example
- 4:16Pattern Generation: Challenges
- 5:06Pattern Generation: Problem Statement
- 5:34Sample Input Logs
- 7:39LogMine: An Automatic Pattern Generator
- 8:55Log Mine: Workflow
- 10:24Pre-Processing: Tokenization
- 11:18Preprocessing: Data-Type Identification
- 12:44LogMine: Pattern-Tree Formation
- 15:06Optimization: Fast Level 1 Formation
- 16:31Example: Distance Calculation
- 17:00Clustering: Example
- 18:44Log Mine: Output Pattern-Set
- 21:22Outline
- 23:05Demo 1: Datacenter Power Usage
- 24:53Demo 2: Custom Application Log
- 26:28Demo 2: Sample Parsing Pattern
- 26:39Demo 2: Sample Parsed Output
- 27:27Summary
related talks
43mLogAn: Large-scale Log Analysis with Small Language Models
39mFast log management for your infrastructure
43mBeyond Logs: Real-Time Anomaly Detection with Machine Learning
27mLog messages processing using NLP tools
01h 00mApplication Logging & Logstash
10mML-based Log Analysis for Faster Debugging
51mCentralized Logging Patterns
42mMake your logs work for you