So many results... Give them to me!
summary
Merged search and analytics on Java with Spring Boot and Elasticsearch handles a 150 million‑document index across ~140 nodes and 140 TB, avoiding a second $30k/month cluster by adding support for full exports. Implementation uses point_in_time plus slicing (needed Elasticsearch 7.15, cluster upgraded from 7.11), falls back to search_after versus the scroll API, and streams results via Spring’s streaming response body. Tweaked thread pools, connection limits and bigger CPU instances, ran merged performance tests, and served multi‑million result exports in about a minute.
AI-written summary · about
chapters (11)
- 0:00Introduction and context
- 1:08Project architecture and costs
- 3:12Merging services challenge
- 4:57Scroll API vs Search After
- 7:05Upgrading Elasticsearch
- 7:49Streaming response choice
- 9:12Performance testing approach
- 10:16Investigating slow queries
- 12:04Scaling the environment
- 13:10Final results and savings
- 14:11Q&A session
related talks
44mSearching through large graphs using Elasticsearch
54mScale Elasticsearch for Your Java Applications. Philipp Krenn. Java Fest 2020 Online
44mElasticsearch at Scale of Billions of Documents
47mAdvanced (Elastic) Search for Your Legacy Application
54mAdvanced (elastic)search for your legacy application
41mTips and Tricks to Scale Elasticsearch for 1M RPMs and Beyond
36mSearch on the fly: how to lighten your Big Data
29mHow (not) to Scale Elasticsearch for Data Analytics!