The Storage Architecture of Intel's Data Management Platform (DMP)
summary
Rack-centric cluster disaggregates storage from compute using LINSTOR + LINBIT orchestrator and Kubernetes with CNI/CSI, exposing NVMe volumes via NVMe-over-Fabric and local Optane via Optane DCPM namespaces. Cluster uses 3-stage Clos, 100 Gb leaf links saturated by four NVMe devices per storage node and 50 Gb compute links saturated by many concurrent volumes; volumes are non-redundant with zone-label placement so stateful services provide redundancy. Performance tests show ~1–1.5 GB/s per Minio bucket and ~60% speedup vs legacy Hadoop for one customer.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 0:40What is the DMP
- 2:33What is De disaggregation
- 3:10Rack Scale Architecture
- 4:20Capacity Gaps
- 5:29The Stack
- 6:52Software Stack
- 8:28Small Organization
- 8:57Linstore
- 12:20Linstore Explanation
- 15:24Linstore Roadmap
- 18:45Network Virtualization
- 19:38Performance Snapshot
related talks
27mSolving inefficiency challenges at Datacenter scale
25mContainer Attached Storage: Open Source Extensions: Kubernetes + Data
29mReliable and Cost-Effective: Open Storage Strategies for Kubernetes
35mEconomics of using Local Storage Attached to VMs on Cloud Providers [I]
28mPersistent Storage with Kubernetes in Production - Which Solution and Why?
37mBuild a Kubernetes Based Cloud Native Storage Solution From Scratch
28mKubernetes Storage Evolution: Enabling High Performance Distributed Datastores
33mPiraeus Datastore - Persistent Volumes for Kubernetes