Lessons Learned from Scaling RabbitMQ in a High-Growth (...)
summary
Company scales RabbitMQ from dozens to millions of customers and processed billions of tasks daily on a message-driven platform; a five-node mirrored cluster eventually triggers a catastrophic 2019 failure so the architecture migrates from one 5‑node mirrored cluster to many standalone brokers per workload, improving performance. Operations rely on metrics (queue growth, consumer utilization), tune heartbeats/TCP keepalive, and limit queues because queue backlogs >1,000,000 triggers emergency. Backpressure strategies include slowing producers, local journaling, TTLs on polling messages, and use feature toggles to shunt to SQS/Dynamo.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:50Product overview
- 6:45High-volume misconfigurations
- 9:12Handling outages
- 10:14Key metrics
- 12:41Performance tuning
- 14:24Back pressure
- 16:51Cluster partitioning
- 18:37Cluster to standalone
- 20:02Key takeaways
related talks
24mAdvanced RabbitMQ Strategies: A case study
40mScaling RabbitMQ
39m101 ways to break your RabbitMQ
32mBuilding distributed system for retail with RabbitMQ
21mWhy does RabbitMQ stop and what can be done about it
19mBuilding a Distributed Data Ingestion System with RabbitMQ
39mReal World RabbitMQ: Exploring Practical Applications
48mA Beginner's Guide to Inter-Service Messaging