When the Cloud Fails: Debugging the "Undocumented"
summary
Migration to a new cloud project triggers a production-only failure that cuts message delivery by about 7% drop for merchants and ~4% for drivers; targeted load tests, tcpdump/Wireshark packet traces, and VM-level metrics isolate cyclic TCP retransmission at 5% on HAProxy. Root cause is a managed load balancer + firewall hitting a hidden connection-tracking table limit; cloud adds a new firewall/connections_tracked metric and documenting the behavior after evidence-based debugging and a whitelist workaround restores delivery.
AI-written summary · about
chapters (8)
- 0:00Introduction MQTT overview
- 4:40Migration failure rollback
- 8:15Root cause HAProxy load balancer
- 12:11TCP retransmissions staging
- 13:16Firewall drops cloud insights
- 16:25Root cause firewall rules
- 17:48Lessons learned observability
- 19:33Q&A
related talks
21mThe Cloud is Someone Else's Network: Debugging the Invisible
09mLightning Talk: Honey, I Broke the Things: Debugging Gray Failures in Production!
28mThe 900-Second Ghost: Debugging Half-Open TCP
20mNetwork Monitor: A Tale of ACKnowledging an Observability Gap
13mHoney, I Broke the Things: Debugging Gray Failures...
36mCloudy With a Chance Of Chaos: Verifying the Resiliency Of Cloud-Native Applications
31mDebugging the Routing Tier
34m5 Things I Learnt from Causing a Cloud Provider Outage