Scaling RGW D4N Caches With Experimental Offloading
summary
RGW offloads D4N caching to external stores to speed bucket-index work: a single-node Redis prototype exists now and the team is building a FoundationDB-backed D4N driver that uses FoundationDB’s total keyspace ordering and native transactions to deliver strong ordering and consistency. Code includes a libfdb C++ wrapper for easier integration. Goals include read-after-write semantics, write‑back caching of bucket listings and metadata, and scaling experiments aimed at handling 10+ billion object indexes and converging with RADOS at production scale.
AI-written summary · about
chapters (7)
- 0:00Introduction to RGW caching
- 1:48D4N cache architecture overview
- 3:13FoundationDB backend for D4N
- 7:28Q&A on bucket indexes and caching
- 10:59Future work and mutable data caching
- 14:09D4N in S3 vectors implementation
- 17:30Library design and Q&A
related talks
22mSolving Everyday Data Problems with FoundationDB
26mImplementing a Resilient Ceph RGW Object Storage with Erasure Coded Data Pool as the Foundation...
37mTechnical Overview of FoundationDB
43mCaching half a billion user transactions
35mSystems @Scale Summer Q&A Week 3
25mDistCache: Provable Load Balancing for Large-Scale Storage Systems with Distributed...
21mDistCache: Provable Load Balancing for Large-Scale Storage Systems with Distributed Caching
12mDirect Access, High-Performance Memory Disaggregation with DirectCXL