Next-Gen Search: How Uber and the OpenSearch Community Built a Cloud Native 3.0
summary
Uber describes moving from a Lucene lambda pipeline to contributions that shape OpenSearch 3.0: a decoupled reader and writer roles pattern with writers flushing sorted segments to S3 and readers pulling them, a Kafka-based ingestor and pull-based ingestion for serverless, and term-based sharding/routing. Production numbers: 50 billion+ searches/day, ~150 clusters, P50 <50ms P99 <150ms and peak ingestion ~100k upserts/sec. Static-rank posting lists, forward-index reranking and subshard concurrent search reduce latency and storage; a cloud-native prototype is underway.
AI-written summary · about
chapters (8)
- 0:00Cloud native OpenSearch intro
- 2:29Uber’s search use cases
- 5:20Current search architecture
- 8:06Uber Eats search case
- 14:27System limits and OpenSearch
- 15:31Cloud native vision
- 19:07OpenSearch roadmap
- 20:32Q&A
related talks
46mDelivering the next wave of innovation with OpenSearch
44mBuilding a Modern Search Engine: The Road Ahead
13mSecuring the Data Frontier: Real-World Strategies With OpenSearch
24mReimagining Search : Scaling OpenSearch Across an Enterprise - Platform C...
31mIntelligent Japanese Search With OpenSearch
33mLessons From Migrating To OpenSearch: Shard Design, Log Ingestion, and UI Decisions
45mManaging a search engine for over 600 million openly licensed media records
29mMastering Search Performance: Advanced Query Insights for Production OpenSearch