Distributed OpenSearch Monitoring at Scale With Apache NiFi and MiNiFi Agents
summary
Architecture using Apache NiFi and MiNiFi deploys a MiNiFi agent per node with a custom NodeStatsProcessor to collect CPU/heap/JVM/I/O/thread‑pool metrics locally and central NiFi collectors with a ClusterStatsProcessor to aggregate and deduplicate cluster-wide stats, forwarding to OpenSearch. Claims linear scalability and sub-millisecond node metrics, runs on 10–50+ nodes, processes millions of metrics daily with HA, minimal overhead, and 99.9% delivery using site‑to‑site and on‑disk buffering.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:26Single-collector monitoring problems
- 3:11NiFi and MiNiFi for monitoring
- 5:40Proposed MiNiFi architecture
- 6:48Fetching OpenSearch node metrics
- 9:04NiFi processing cluster metrics
- 11:40Resilience and back-pressure
- 13:06Why architecture works
- 14:12Deployment strategy
- 15:39MiNiFi agent options
- 16:41Implementation lessons
- 17:25Key takeaways
related talks
33mAccelerating OpenSearch Service Operations With an Operation Agent
44mNetdata: Open Source, Distributed Observability Pipeline – Journey and Challenge
33mNetdata: Open Source, Distributed Observability Pipeline – Journey and Challenges
32mUnleash Your Cluster's Potential with OpenSearch Benchmark
46mOperational Logs and Event Management for HPC S...
36mAutoscaling Search and Analytics in Kubernetes: The OpenSearch Operator Advantage
52mOperating OpenSearch at Scale: Lessons From Managing 10,000+ Clusters Across...
41mMonitoring Solr at Scale