RabbitMQ has a problem - Let us tell you about it
summary
Deploys a three‑node RabbitMQ cluster on k3s and wires prometheus + grafana + alertmanager to fire Slack alerts; a pending‑pod failure is diagnosed as an accidental 1 TB memory request and fixed by changing it to 1 GB so pods become Ready and alerts resolve. A perf‑test client reproduces queue growth to 50,000 messages (10 KB each → 500 MB) and triggers a predicted disk‑watermark alert; the cluster operator can resize persistent volumes live with minimal downtime. Also uses Loki json logging and cluster operator PV resize to alert on crashes with stack traces.
AI-written summary · about
chapters (9)
- 0:0002:10 - Introduction
- 2:1110:29 - Demo: I know nothing. What do I do?
- 10:3016:10 - 2nd Problem
- 16:1117:22 - Potential Fix
- 17:2318:59 - Our Fix to the Problem
- 19:0023:54 - Past Problems
- 23:5525:28 - Summary
- 25:2929:49 - RabbitMQ 3.9
- 29:5037:01 - QnA
related talks
26mDeploy and Monitor a RabbitMQ Cluster on Kubernetes
36m101 Ways to Crash Your Cluster
37mConfidence with Chaos for your Kubernetes Observability
29mKubernetes Failure Stories and How to Crash Your Clusters
35mFrom New Cluster to Insight. Deploying Monitoring and Logging to Kubernetes
45mDecentralizing SaaS Monitoring Using Open-Source Tools and Standards
46mFollow the RabbitMQ
42mMaking Resilient, Self-Adaptive, And Autonomous Kubernetes Clusters