Connection Storms: Detecting and Defusing Outages Before Valkey Crashes
summary
Connection storms are sudden floods of TCP handshakes that overflow the kernel conntrack table and the server accept queue (syncq/backlog), often after deploys or network flaps, causing timeouts and missing metrics. Practical fixes: cap client pools or use multiplexing (Glide) instead of N sockets, implement exponential backoff with jitter, monitor socket stats for SYN/accept queue lengths, raise somaxconn/TCP backlog and watch cloud limits (PPS, bandwidth, NAT table). These changes measurably prevent outage cascades.
AI-written summary · about
chapters (8)
- 0:00Introduction to connection storms
- 1:05What is a connection storm
- 2:32Client-side connection management
- 4:45Reconnection strategies
- 7:15Server-side monitoring
- 8:41Kernel-level metrics
- 12:42Four key takeaways
- 13:05Q&A
related talks
20mNetwork Monitor: A Tale of ACKnowledging an Observability Gap
30mNoisy Neighbors, through Networking
31mOn Paravirualizing TCP: Congestion Control on Xen VMs
28mThe 900-Second Ghost: Debugging Half-Open TCP
32mSafe Client Behaviour
31mWhat Breaks Our Systems: A Taxonomy of Black Swans
53mResiliency and Availability Design Patterns for the Cloud
51mHearts of Darkness: A Spring DevOps Apocalypse