Debugging the Routing Tier
summary
Symptoms: frontend GETs succeed but immediate PUTs fail; debugging uses tcpdump and Wireshark to show HAProxy sending a FIN while the Java HTTP client silently reuses the closed socket, revealing a timeout mismatch (HAProxy 0.5s vs client keepalive 5s) as the root cause. Second case: intermittent API hangs and some gorouters show high memory; pprof and flamegraphs point to heavy 404 logging causing disk write lock contention; isolating the VM to an empty hypervisor removes the issue.
AI-written summary · about
chapters (17)
- 0:00Start
- 0:35Overview
- 1:56Routing Tier Components
- 5:23Problem Statement
- 7:36The Architecture
- 8:36Steps
- 8:58Application logs
- 9:52What is tepdump?
- 10:23TCPDUMP
- 10:52What is wireshark?
- 13:57Hypothesis 1
- 15:22Hypothesis 2
- 15:41Another Tidbit
- 16:05Hypothesis #2
- 17:48Lessons learned
- 19:26Gather formation
- 30:56References
related talks
21mImproving L4 Routing Consistency
36mWhat Is Happening with My Connections? Why Are They Closed?
25mScaling Up, One Network Bottleneck at a Time
43mDeploying and Debugging HTTP/3
31mLayer Four and Three Quarters: Fantastic Quirks and Where to Find Them
34mHyperRouter: Lessons Learnt from Building an L4 Load...
28mThere's a Bug in My Service Mesh! What Do You Do When the Mesh is At Fault?
58mTelco; Networking; Architecture; How to debug; Tips and Tricks