DedupSearch: Two-Phase Deduplication Aware Keyword Search
summary
DedupSearch scans each unique chunk once in a two-phase: physical then logical pipeline, recording exact matches keyed by chunk fingerprint and saving boundary substrings so hits split across adjacent chunks are recovered. Tiny one-character partial matches are spilled to an on-disk tiny-result index to bound memory. Evaluation on English Wikipedia and many Linux backups shows practical gains — matching up to 10x faster with up to 99.9% less data read compared to naive file-by-file search.
AI-written summary · about