Getting GPUDirect to Work with Broadcom NICs
summary
Implemented GPUDirect RDMA between a Broadcom NIC and NVIDIA GPU using DPDK by changing three layers: application, gdrcopy, and the Broadcom PMD. Key actions: modified Broadcom PMD to support header/payload split, used P2P_get_pages mapping to pin UVM memory, extended gdrcopy with a gdr_copy_tma API to return real DMA addresses, and passed those DMA addresses into the NIC. The pipeline delivers packets directly into GPU buffers, CUDA manipulates MACs to prove processing, and end-to-end DMA transfer works.
AI-written summary · about
chapters (5)
- 0:00Introduction and background
- 1:05What is GPU Direct RDMA
- 3:34Key components and architecture
- 8:30Memory mapping and DMA address translation
- 11:02Proof of concept and results
related talks
26mGPUDirect + SDR: How to Move One Billion Samples per Second over PCIe
20mPassthrough/Headless GPU Gets Ahead
24mImproving GPU Utilization and Accelerating Model Training with Scheduling Framework and NRI
33mGPU Para-Virtualization on Xen
16mTurning PCIe Hints into Cache Hits: Enabling Smart Data Cache Injection in Linux
30mBeyond the DC Walls: Building a Nationwide GPU Fabric With KubeVirt and Wide-Area…
34mDeclarative Device Virtualization: Orchestrating GPUs & Hardware in Cloud Native Environments
17mVDUSE - vDPA Device in Userspace