Stateless Search: Decoupling Storage and Compute for Search Engines
summary
An overview of Quickwit, an open-source serverless search engine that separates storage (object stores like S3) from compute. It explains how to perform sub-second searches over data kept on object storage, the performance and operational constraints this imposes, and demonstrates scaling indexing and query workloads on a Kubernetes cluster handling terabytes of data.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 4:46Log search specifics
- 9:31Shared Nothing architecture
- 19:59Shared Disk architecture
- 27:42Shared Disk pros and cons
- 31:49S3 optimizations
- 45:15Demo and conclusion
related talks
35mHighly Available Search at Shopify
34mBuilding a WebScale Search Engine with a Data Driven Architecture
34mFunnel Rocket: A Serverless Query Engine
26mHigh-Performance Full-Text Search on Apache Iceberg Using Lucene-Integrate...
33mDelivering business intelligence at hyperscale
46mWings Stay On or Replacing On Site Search at Wikipedia
46mKEEPING UP IN A DATA-INTENSIVE WORLD WITH SINGLESTORE
36mThe Imitation Game: Encapsulate Your OSB Workload at Scale