Autoscaling Elasticsearch for Logs on Kubernetes
summary
An operator autoscaling Elasticsearch/OpenSearch for logs changes index templates to alter primary shard count, changes shards in template and rotates indices by size (rule of thumb ~10 GB per shard) so new nodes immediately help indexing. Demo uses a modified Zalando es-operator, ties scaling to disk thresholds and min/max replica settings, and successfully scales a cluster from 2→4→2 nodes while updating ISM/ILM policies. Proof of concept is functional but not production-ready and is untested at very large shard sizes.
AI-written summary · about
chapters (13)
- 0:00Start
- 1:16Agenda
- 2:39What Does the Operator Need To Do in Order to Auto Scale
- 2:49Using Time-Based Indices
- 3:20Time-Based Indices
- 6:15Rotate Indices by Size
- 7:01Index State Management
- 8:53Change the Index Template
- 13:42Elastic Cloud in Kubernetes
- 15:16Demo
- 28:00What Matrix Are Your Monitoring in Order To Know When To Add More Nodes or Scale More Less Nodes
- 28:24Disk Thresholds
- 31:36Large Shards
related talks
32mAutoscaling Elasticsearch for Logs on Kubernetes
36mAutoscaling Search and Analytics in Kubernetes: The OpenSearch Operator Advantage
14mAutoscaled K8s Observability with the Elastic Operator
39mOperating OpenSearch at Scale With Kubernetes Operators: Lessons Learned and Community Insights
29mScaling Search Clusters with Apache Solr and Kubernetes
52mScale Your Metrics with Elasticsearch
32mScaling Shopify's Search: Enhancing Elasticsearch Resilience With Kubernetes and KE...
52mOperating OpenSearch at Scale: Lessons From Managing 10,000+ Clusters Across...