Operational Logs and Event Management for HPC S...
summary
HPCM deploys a monitoring pipeline that streams sensor and syslog data via Fluent Bit into Kafka, then uses Logstash to index into OpenSearch and OpenSearch Dashboards for visualization; operational alerting uses a YAML-based wrapper around OpenSearch Alerting to route to Alertmanager, email or Slack and trigger automated remediation with a first-responder webhook. Production scale is a 10-instance OpenSearch cluster on 10 nodes ingesting ~295 GB/day syslog (≈2 TB for 7 days), and this stack runs successfully on Aurora (≈10,624 compute nodes, 21,248 CPUs, 63,744 GPUs).
AI-written summary · about
chapters (30)
- 0:00HPC monitoring with OpenSearch
- 1:03Our HPC systems role
- 2:50High performance computing
- 7:05HPC infrastructure scale
- 9:07HPCM supercomputer manager
- 10:30HPCM monitoring features
- 11:56HPC systems monitoring
- 12:39HPC monitoring stack
- 13:20Why OpenSearch for HPC
- 14:44OpenSearch syslog setup
- 15:06Syslog dashboard flow
- 16:08Fluent Bit log config
- 17:57Logstash log pipeline
- 18:59OpenSearch cluster config
- 20:03Index lifecycle policies
- 20:44OpenSearch alerting challenges
- 23:35Custom OpenSearch alerting
- 27:08Syslog alert example
- 30:19Alert webhook beautifier
- 31:43Alert Manager visualization
- 32:48Webhook automated actions
- 33:08Aurora supercomputer overview
- 34:59Aurora hardware details
- 36:02Monitoring hardware with OpenSearch
- 37:04OpenSearch node specs
- 37:47Log retention policies
- 38:29Power off event alerts
- 39:09OpenSearch benefits feedback
- 39:50Future analytics vision
- 40:39Q&A
related talks
35mFrom Chaos To Clarity: Centralizing Observability With OpenSearch at Scale
37mBuilding Modern Observability Architecture With OpenSearch, OS Ingest...
21mKubernetes Under the Magnifying Glass: Observability With OpenSearch
33mAccelerating OpenSearch Service Operations With an Operation Agent
20mHow We Implemented a Near Real-Time Logging Solution Using Fluent Bit And OpenSearch
30mBuilding a Large-Scale, Streaming-Based Logging and Monitoring Solution
27mIngesting 6.5 Tb of Telemetry Data Daily Through Open Telemetry Protocol and Collectors
22mFrom Telemetry to Insight: Building AI-Powered Observability Pipelines with OpenSearch