From Manual to Managed: Prometheus Agent Deployment at Scale
summary
The team deploys Prometheus agents four ways and explains tradeoffs: Helm + ArgoCD for seconds-scale installs but requires Kubernetes and cluster management consent; an Ansible package that bundles binaries and templates for air-gapped on‑prem installs but lacks reconciliation; Cluster API + Kamaji to create one or two hosted nodes per customer so Kubernetes control plane stays central; and a Go templating installer used on Raspberry Pi for one‑off installs. Networking uses Cloudflare geo load balancing with per‑customer tokens and MARB ID; demos mostly work but live demo flakiness is noted.
AI-written summary · about
chapters (9)
- 0:00Prometheus agent challenges
- 1:26Why automate agent install
- 2:10Networking with Cloudflare
- 4:02Helm GitOps Argo CD
- 5:50Ansible on-prem deployment
- 10:20Cluster API Kamaji nodes
- 18:34Golang installer non-K8s
- 25:46Deployment strategies future
- 26:30Q&A: Agent vs exporter
related talks
24mDeclarative Monitoring pipelines with Prometheus
32m“Gluing it all together” aka building a platform around Prometheus
35mStateless Collectors For Stateful Data: Scaling Prometheus As a Node Agent
46mProgressive Deployments with Consul Service Mesh
08mObserving Agentic/MCP Traffic & Keeping Costs Low
32mAdapting Brownfield Deployments to Cloud-Native Technology
37mDeploying applications to Kubernetes: never send a human to do a machine's job
27mPrometheus: The Unsung Heroes