Reasoning about performance (in the context of search)
summary
Shows a concrete claim and measurement: a search design called BitFunnel uses Bloom‑filter–style indexes, back‑of‑the‑envelope math, and simulations to estimate throughput and latency before coding. It buds from the problem of 10PB of data into concrete costs, shard counts and a surprising estimate of about 3900 QPS on a single server for a 10M‑doc shard, describes hierarchical bloom filters, ingestion rates, where the approach breaks, and reports it actually works in production.
AI-written summary · about
chapters (15)
- 0:00Thinking about performance
- 0:11Search: a case study
- 1:34Coding feels like real work
- 3:03What's the actual problem?
- 4:26$2000 for 128GB server
- 10:17Search Algorithms
- 12:0810s of billions per shard!
- 15:43Probability of false positive?
- 16:56Linear cost Exponential benefit
- 18:59How do we estimate perf?
- 21:46Expected performance?
- 23:03Ingestion
- 23:53Hierarchical bloom filters
- 26:08Conclusions?
- 28:29Conclusions!
related talks
01h 00mHyper Speed: When Big Data Blooms
43mIn Search of Lost Performance
30mCaching for Speed: Practical Performance Optimization in OpenSearch & Lucene
25mHigh on Cardinality, Low on Memory
19mEnhance Search Performance At Scale with
28mBuilding a Cutting Edge Search Engine in Go
48mAnalytics Architectures on Microsoft Fabric
37mWhen Probably is Good Enough