Over a Billion Requests Served per Day: Ensuring Everyone Is Happy with Our Ceph Clusters’...
summary
Bloomberg runs over a billion S3 requests per day across four Ceph sites (~65 PB, ~4,500 OSDs, ~50k buckets) and builds a software-defined QoS on the RGW that enforces per-second verb limits and aggregate bandwidth caps at access-key and bucket-level throttling. The stack uses HAProxy with Lua and C patch, Redis for short-lived epoch counters, a syslog forwarder, and a policy-generator that fans violations to gateways. Enforcement happens via HTTP 503s and millisecond sleeps for throttling, with node-level stick-table fallbacks; the system runs in production and yields stable, consistent user limits.
AI-written summary · about
chapters (7)
- 0:00Start
- 19:12Our Software Stack
- 20:00Parsing network traffic
- 20:50Aggregate Stats
- 22:00Calculate violations through Policy Generator
- 25:51Targeted & Dynamic QoS
- 27:12Rely on a backup plan
related talks
15mAn Open Source Distributed Rate-limiting Service for Ceph-object and Other HTTP APIs
38mDelivering Enterprise-Grade Object Storage: Bloomberg’s IAM Journey in a Multi-Tenant Arch...
24mScaling to Billions of Internet Observations per Day
37mIntroducing Ceph Mgmt-gateway: Unified Access, OIDC-based SSO, and HA for Manageme...
29mSecuring Data in Motion and at Rest: Quantifying End-to-End Encryption Perfo...
36mCeph Notification at Scale: Notification V2, an Enterprise-Level Feature
25mServerless Osquery Backend and Big Data Exploration
23mSMB Keeping - Clients, Clusters & Clouds