Tuning Elastic on Kube...
summary
ING runs Elastic on OpenShift with Portworx across four clusters (44 workers, 96 CPUs per node, ~65k pods) and 2.7 PB of storage; indexing rate drops and right timeouts trace to slow storage despite stable cluster metrics. Troubleshooting uses fio and Elastic stress tests, reproduces issues in an isolated environment, and benchmarks storage classes. Fixes: upgrade to Portworx 3.1.0, remount volumes with replica 1, add journal device with autojournal, enable no_discard/out_of_stream and opscon high; write throughput nearly doubles while keeping RAID5.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 1:03ING and Kubernetes platform
- 6:47Elastic stack on Kubernetes
- 8:37Performance issues in Elastic
- 10:23Storage bottlenecks
- 11:49Troubleshooting methods
- 13:55Baseline tests and scaling
- 15:01Portworx deep dive
- 17:29Root cause analysis
- 18:31Production fixes
- 19:36Monitoring pitfalls
- 23:52Lessons learned
- 24:55Closing and Q&A
related talks
44mPerformance and Scalability Tuning Kubernetes for OpenShift and Docker
29mMoving & managing k8s persistent data across clusters and cloud
11mand Amazon EKS: A great way to run Kubernetes applications (PRT006)
34mPanel: Improving and Managing Kubernetes at Scale
44mFinOps at scale: Lessons from scaling Kubernetes workloads
31mLoad Testing Kubernetes: How to Optimize Your Cluster Resource Allocation in Production [I]
35mHA and Data Mobility for persistent Volumes
02h 26mKubernetes Performance Tuning Workshop