One Port, Many Gateways: Scaling RGW for Uptime & Production Resilience
summary
Bloomberg enables multiple Ceph RGW processes to bind to the same port using the Linux SO_REUSEPORT kernel feature, changes in the RGW startup code (pull request already merged upstream), and systemd/Ansible automation that starts a new instance before draining the old one. At Bloomberg this cuts a 3–4 hour rolling-restart window to near-zero, reduces client 500 errors by ~500x, consolidates three per-node gateways onto a single port (8000), and uses a graceful_shutdown timeout = object_size / bandwidth heuristic.
AI-written summary · about
chapters (8)
- 0:00Introduction and background
- 2:50Problem with system degradation
- 6:23Solution with port reuse
- 11:22Implementation steps
- 14:33Scaling with single port
- 15:58Graceful shutdown settings
- 17:25Benefits and summary
- 18:47Q&A on rolling restarts
related talks
18mTGW: Operating an Efficient and Resilient Cloud Gateway at Scale
43mMaximizing the Value of Your Rados Gateway with Ingress Strategies
34mProxying Rabbits For Stability & Reliability
26mSIG Network Intro and Updates
26mRedesigning Ingress: Docker’s Transition To the Next-Gen Gateway API
44mNVMe-Over-Fabrics Support for Ceph
38mCeph NVMe-of Road Map
29mSimplifying the Networking and Secur...