Resilient Segment Replication With Adaptive Refresh
summary
Segment replication causes huge checkpoints that trigger major page faults; implementing byte-sized commits (multiple Lucene commit points per replication interval) plus an adaptive refresh policy that picks the largest commit below a byte threshold reduces thrashing. Replicas download only files listed in commit metadata and refresh in up to four hops in tests with artificial cross-region delays; one forced-latest-commit edge case still spikes faults. Changes land in Lucene 10.3 and are optional to enable in downstream systems.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 1:24Search system fundamentals
- 5:17Segment replication in search
- 7:42Benefits over document replication
- 9:53Payload size vs latency
- 13:47Large payload causes
- 15:52Payload absorption by searchers
- 18:22Memory thrashing from payloads
- 20:30Byte-sized commits insight
- 21:35Multiple commit points
- 24:25Adaptive refresh strategy
- 26:31Simulation testing
- 27:58Downloading only necessary files
- 30:48Applying to OpenSearch
- 33:40Trade-offs and considerations
- 39:00Q&A
related talks
22mDesigning Concurrent Distributed Sequence Numbers ...
18mWho's Afraid of Uncorrectable Bit Errors? Online Recovery of Flash Errors with Distributed Redundancy
20mExploiting Commutativity For Practical Fast Replication
48mReplication as a First-Class Citizen: The Art of Global Resilience
45mOK Log: Distributed and Coördination-Free Logging
19mBuilding a Performant Configuration-Store Layer
18mLeveraging Multiple Regions to Improve Site Reliability:
12mFault-Tolerant Replication with Pull-Based Consensus in MongoDB