CLP: Efficient and Scalable Search on Compressed Text Logs
Talkat 15th USENIX Symposium on Operating Systems Design and Implementation (OSDI '21) · 2021-07-14..07-16·
summary
A system called CLP losslessly compresses text logs and stores 14 TB as 328 GB (2.26% of original) and reports an average compression ratio of 32.2 versus gzip 16.38, Splunk 2.86 and Elasticsearch 1.75. CLP ingests logs over 13x faster than index-based tools, scales to petabytes, supports wildcard and regex search without full decompression, pipes operators for parsing/unique aggregation, exposes a FUSE filesystem and shows search performance 4.2× faster than Splunk, 1.3× faster than Elasticsearch, and 7.8× faster than ripgrep.
AI-written summary · about
chapters (6)
- 0:00The problem with log search
- 4:20Introducing CLP
- 6:14How CLP works
- 8:35Search challenges
- 9:54Performance evaluation
- 13:50Conclusion and future work
related talks
09mLogCrisp: Fast Aggregated Analysis on Large-scale Compressed Logs by Enabling...
16mμSlope: High Compression and Fast Search on Semi-Structured Logs
16mPolarStore: High-Performance Data Compression for Large-Scale Cloud-Native Databases
30mData @Scale Seattle- Sundaram Narayanan
27mRealise the value of data with Splunk
25mPetabyte-scale Log management in Postgres using Bagger
27mData @scale Seattle - Yucheng Low
41mUnifying Diverse Logs in Big Data Systems for Seamless Analysis and Action With OpenSe...