μSlope: High Compression and Fast Search on Semi-Structured Logs
Talkat 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI '24) · 2024-07-10..07-12·
summary
μSlope losslessly compresses semi-structured logs by extracting and storing schema trees once, grouping records with the same schema into encoded record tables, and encoding columns by type (variable dictionary, timestamp patterns, binary numbers). It uses a merged Parts Tree and schema map to find matching schemas so queries run with minimal decompression. Evaluation on 21 datasets reports 21.9:1–186.8:1 compression (2.34× better than Zstandard) and search up to 5.77× faster than other SSDMSes; implementation is open source.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 0:33Challenges of log management
- 4:31Limitations of prior systems
- 6:49System design observations
- 8:16Data encoding and structures
- 10:56Architecture and search
- 12:58Evaluation and results
- 14:48Conclusion
related talks
15mCLP: Efficient and Scalable Search on Compressed Text Logs
16mPolarStore: High-Performance Data Compression for Large-Scale Cloud-Native Databases
09mLogCrisp: Fast Aggregated Analysis on Large-scale Compressed Logs by Enabling...
27mData @scale Seattle - Yucheng Low
41mUnifying Diverse Logs in Big Data Systems for Seamless Analysis and Action With OpenSe...
37mFrom Inverted Index to Columnar Vectorized Execution Search
51mCase Study of Moving from CRUD to Log Based State Management
48mDesign of data-intensive applications