Building a real-time news search engine #bbuzz
summary
Bloomberg replaces a legacy backend with a custom real‑time news search built on Solr/Lucene, indexing ~500 million stories, ingesting peaks of 500 stories/sec, and serving ~10 million searches/day with average responses ~180 ms and results available in 125 ms. Changes include new Lucene query types, tag relevance normalized via tf‑idf, time‑sorted segments, SSD merge tuning, GC and JVM fixes, query circuit breakers, and a learning‑to‑rank framework; the system scales across hundreds of shards and thousands of cores and runs in production.
AI-written summary · about
chapters (17)
- 0:00Introduction to Bloomberg
- 2:10Search ecosystem overview
- 3:49Scale and challenges
- 6:13Migrating to Solr/Lucene
- 8:11Development methodology
- 8:49Data model and query syntax
- 13:05Architecture and infrastructure
- 15:18Implementing search parsing
- 17:36Handling tag relevance
- 20:36Performance optimization
- 23:24Java and JVM tuning
- 25:54Ensuring system stability
- 28:25Managing race conditions
- 30:45Preventing poisonous queries
- 34:03Data consistency challenges
- 35:18Advanced features
- 36:20Future initiatives and Q&A
related talks
45mRealtime Search at Twitter
36mSearch on the fly: how to lighten your Big Data
49mReplatforming Online Search for a Major High Street Retailer
57msolr-ruby: The Best Open Source Search Engine + Ruby
19mEnhance Search Performance At Scale with
38mIntroduction to Search
26mScaling LinkedIn’s search infrastructure
36mCrateDB: A Search Engine or a Database? Both! #bbuzz