Duhu: Shared Disaggregated Memory for Distributed Data Processing Frameworks
Talkat 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI '26) · 2026-07-13..07-15·
summary
A system that replaces per-node object copying with a segmented, owner-assigned SDM layout and an in‑SDM ring‑buffer RPC channel, implemented on a prototype CXL-attached memory pool and integrated with Ray. It uses a read‑only owner/borrower model, a get‑ref protocol that updates per‑segment metadata and flushes caches, and a three‑step recovery that replays idempotent RPCs from SDM logs. Evaluation on an emulated 16‑node setup (10 GB/s pool) shows up to 3.39× job completion time improvement on a multi‑stage shuffle and 2.8–4.2× shorter object get latency.
AI-written summary · about
chapters (7)
- 0:00Duhu introduction
- 2:28Duhu architecture
- 4:13Duhu design techniques
- 8:31Duhu coordination channel
- 10:20Duhu failure handling
- 11:48Duhu performance
- 16:02Duhu future directions
related talks
12mDirect Access, High-Performance Memory Disaggregation with DirectCXL
19mFast Distributed Transactions for RDMA-based Disaggregated Memory
24mShray: an Owner-Compute Distributed Shared-Memory System
12mConcordia: Distributed Shared Memory with In-Network Cache Coherence
13mDMTree: Towards Efficient Tree Indexing on Disaggregated Memory via Compute-side...
14mPD3: Prefetching Data with DPUs for Disaggregated Memory
16mDisaggregation and the Application
16mDRust: Language-Guided Distributed Shared Memory with Fine Granularity, Full...