Elasticsearch index management for paas style logging
summary
NELO ingests over 3 billion logs per day (1.6 PB stored) and hits a mapping explosion where per-index mappings grow and block cluster state updates; 3,000 projects create thousands of mappings daily. The index manager pre-creates aliases, snapshots to HDFS, moves indices across hot/warm clusters, and enforces dedicated vs common indices—projects above ~50M/day get dedicated indices while others share a common index. Shard counts are dynamically sized from recent per-project ingest and a 20 GB shard target guides placement and retention enforcement.
AI-written summary · about
chapters (18)
- 0:00Start
- 2:59Design Considerations 1: Heterogeneous Logs
- 4:16Design Considerations 2: Scale
- 5:12Architecture of NELO
- 7:20Cluster Architecture V1.0
- 8:11Index Model V1.0
- 11:42Problem 1: Mapping explosion
- 12:31Imbalance of shard size
- 14:08Impact of big projects
- 14:30Retention time
- 16:49Type conflict
- 19:21Elasticsearch Cluster Architecture V 2.0
- 20:39Index Model 2.0
- 22:24Indexing/Searching V 2.0
- 24:05What Index Manager is?
- 24:40Scheduled & Real-time
- 27:36Architecture with Index manager
- 36:23Backup
related talks
40mElasticSearch: Big Data, Search and Analytics
30mData @Scale Seattle- Sundaram Narayanan
01h 00mELK Stack - All You Need for Search In Your Applications
42mGoing Elastic
14mLogging Apache Spark - How we made it easy
29mRealtime Analytics Unleashed: Unlocking the Potential of Data at Scale
41mUnifying Diverse Logs in Big Data Systems for Seamless Analysis and Action With OpenSe...
19mLarge Scale, Solr & HBase Performance Monitoring