MetaRoCE: From Spec to NIC to Open Source
summary
MetaRoCE implements a receiver-driven, multipath RDMA stack with a software reference, compliance suite, and an AMD P4 programmable NIC implementation; experiments run on an 8x100G multiplane fabric show subsecond failure detection and full recovery under 10 seconds, per-path congestion window capped at 256 with scale-across using 30–50 instantiated paths to reach high RTT links, selective ACKs keep all-reduce performance near line rate even at 1-in-1,000 packet loss, and MetaRoCE components are being released at OCP including the spec, reference code, and tests.
AI-written summary · about
chapters (10)
- 0:00MetaRoCE introduction
- 1:08Development process
- 3:13Hardware prototyping with AMD P4
- 4:59P4 pipeline features
- 8:10Congestion-aware load balancing
- 9:15Scale out and connections
- 11:03Receiver-based congestion control
- 12:26Selective acknowledgment
- 13:29Validation and results
- 15:36Open sourcing MetaRoCE
related talks
21mScaling RoCE Networks for AI Training
18mRoCE BALBOA: Service-Enhanced RDMA Offload Engine for Data Center SmartNICs
20mRDMA at Cloud Scale: The OCI Experience
19mBring RoCE (RDMA over Converged Ethernet) Capability to Virtio-net Device
12mPreview: Networking
23mMulti-Path Transport for RDMA in Datacenters
15mFlor: An Open High Performance RDMA Framework Over Heterogeneous RNICs
18mFileMR: Rethinking RDMA Networking for Scalable Persistent Memory