War stories from the CloudAMQP support
summary
CloudAMQP support shares concrete failure modes, fixes and numbers: handling 5,500 support tickets/year, reproducing a backlog by publishing 2 million messages to compare quorum queues with and without memory limits, and debugging a 50–60k MQTT connection case that led to changing the credential obfuscation default and tuning TCP buffers. Recommendations include quorum queues recommended (less config pain), watch the unroutable metric for missing bindings, a PG2/ETS federation bug fixed in 3.8.10, and a new MyraMQ edge bridge in Crystal to offload massive MQTT connections.
AI-written summary · about
chapters (10)
- 0:0003:07 - Introduction
- 3:0605:56 - My messages are lost
- 5:5707:59 - I keel loosing my messages
- 8:0010:37 - My Quorum Queues took all my memory
- 10:3611:48 - Does it affect consumption?
- 11:4913:20 - My RabbitMQ MQTT server is running out of memory
- 13:2118:46 - What is using all that RAM?
- 18:4722:17 - Some ETS tables are slowly eating my memory
- 22:1826:36 - Statistics update
- 26:3741:16 - QnA
related talks
46mFollow the RabbitMQ
28mThe 900-Second Ghost: Debugging Half-Open TCP
21mWhen the Cloud Fails: Debugging the "Undocumented"
39m30 minutes of Failure
30mAbove the Clouds: Investigating Systems from 32,000 ft.
36m101 Ways to Crash Your Cluster
26mApache ActiveMQ and the coming features around cloud and management
53mBottlenecks in Serverless Applications