A Universal Document Index: How We Index Billions of Documents

on YouTube ↗45m

" summary

Explains how Elasticsearch works and scales and why its built-in features are insufficient for all search and filtering scenarios. Describes the motivation for building a single universal index to cover those scenarios and the engineering decisions used to implement that index for indexing billions of documents.

AI-written summary · about

" chapters (7)

1. 0:00Introduction
2. 4:03Need for universal index
3. 7:55Elasticsearch internals
4. 12:57Indexing challenges
5. 22:50Filtering optimization
6. 27:03Summary and next steps
7. 28:28Q&A

" related talks

• 16mWhy we built ElasticSearchShay Banon · dotScale 2013
• 55mFull-Text Search DemystifiedPhilipp Krenn · JavaDay Lviv 2016
• 10mGetting Your Data into Elasticsearch: Let Me Count the WaysJay Miller · PyOhio 2021
• 44mSurviving ElasticsearchVitaliy Kharytonskiy · Highload fwdays'18
• 24mIndexing Office Documents with the Elastic Stack and FSCrawlerDavid Pilato · Voxxed Days Luxembourg 2022
• 44mElasticsearch at Scale of Billions of DocumentsIgor Kupczyński · Confitura 2015
• 33mDriver for Data-Driven ChangeKristoffer Axelsson · Azure Cosmos DB Conf 2022
• 48mIndexing Your Office DocumentsDavid Pilato · Big Data Conference Europe 2022