GreenOps at Scale: The 50TB Memory Diet for a Petabyte Platform
summary
Claims a 50TB memory saved across a petabyte logging platform by switching hot nodes from AWS EBS to local NVMe, coordinating migrations with OpenEBS + LVM, fio microbenchmarks and a small-scale reproducer of real queries. Results show IO wait as the root cause, a 48% ERU reduction to avoid Elastic license penalties, P95 query latency ~9x better, ~38% lower AWS bill, and a zero‑downtime cluster migration using replica reshuffling and S3 snapshots.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 2:58Pet to cattle infrastructure
- 7:36Scaling Elasticsearch clusters
- 10:24IO bottlenecks in system
- 12:1042% memory reduction challenge
- 14:16Benchmarking storage performance
- 17:49Local disk vs EBS performance
- 20:39Optimizing for SLA and P95
- 22:06OpenEBS implementation
- 23:52Performance gains and savings
related talks
19mEnabling Big Memory workloads to run
24mCloud Workloads At Petabyte Scale
01h 06mBest practices when accessing Big Data or any other data!
47mExploring Linux Memory Usage and IO Performance in Cloud Native Applications
17mBoosting Cost Efficiency by Cost Aware Architecture
47mScaling FinOps with a holistic approach to resource optimization
30mFrom Cloud to On-Prem - Capacity Planning and CPU Analysis
41mFinOps and Cloud Cost Optimization