Monitoring Nomad with Prometheus and Icinga
summary
Builds a working pipeline that wires Nomad -> Consul -> Prometheus -> Icinga2 and Grafana, using Traefik for dynamic ingress and Nomad job metadata (example tag traffic.enable=true) to expose services. Prometheus scrapes with retention 30s and disabled hostnames; health and metrics checks run every 10s, sync job runs every 10 minutes with fast 10s polling. Includes a demo nginx blog job (six failed deploys before one passed) and a custom Consul-import plugin that automatically creates Icinga hosts and services. Visualization uses Grafana dashboards/plugins to produce a Vizceral-like view.
AI-written summary · about
chapters (24)
- 0:00$ whoami
- 0:58Today's Schedule
- 1:23Traefik
- 3:05ssl ingress
- 4:13dynamic ingress
- 5:27Good neighbor
- 5:53Nice Dashboards #4475
- 7:28Consul Services
- 8:23Consul-icinga Exit Codes
- 9:00Consul metrics for all
- 9:30Nice Dashboards #2351
- 10:36Nomad metrics for all
- 10:58Nice Dashboards #6281 #62
- 11:17Lets deploy our blog
- 11:53Service Definition
- 12:54Consul UI
- 13:31Nomad UI
- 14:20Static vs Dynamic Problem
- 15:08Query Consul
- 15:54Adding a sync source
- 16:45Adding a sync rule
- 17:25Add sync properties
- 17:57Plumbing it together
- 19:03Icinga Business Processes
related talks
22mAdding Observability to HashiCorp Nomad Applications with Grafana
25mAutoscaling Your Nomad Jobs Using SRE Insights
46mProgressive Deployments with Consul Service Mesh
25mInspecting Nomad Events Using Grafana Loki
30mAutoscaling with HashiCorp Nomad
13mAutomate an Observable Runtime with HashiCorp and Ansible
24mDeclarative Monitoring pipelines with Prometheus
24mPractical Kubernetes Monitoring with Prometheus