Search in the Cloud: separation of compute and storage
summary
Modified SolrCloud stores one copy of each shard on shared object storage (S3) and makes JVM nodes stateless caches, so nodes can be added or removed without data movement. Indexing writes new segment files locally, uploads them to the zero store and does a conditional ZooKeeper metadata swap to prevent concurrent overwrites. Implementation adds a zero replica type, is available as SIP-20 branch and runs in production at Salesforce scale. Drawbacks: first queries may fail or incur high fetch latency and commits per batch add overhead.
AI-written summary · about
chapters (7)
- 0:00Introduction and motivation
- 3:50Solr Cloud architecture
- 14:35Zero replica implementation
- 21:44Race conditions and updates
- 27:02Query handling and benefits
- 30:33Drawbacks and future vision
- 37:07Q&A session
related talks
44mOpenSearch’s Quest for Disaggregated Storage: The Strangler Fig Patter...
30mUsing Solr unconventionally to serve 26bn+ documents
34mEvolution of Slack Search
34mArchitecting Solr indexing pipelines in Google Cloud Platform
20mBuilding An Elastic Query Engine on Disaggregated Storage
41mThe New Generation Of Data Stores
22mMulti-Purpose Cloud Filesystem
28mCloud The Final Destination