Machine Learning Inference in DPDK
summary
DPDK adds a new mldev spec and a userspace library delivered as libmldev (in DPDK 23.1) to configure PCI and virtual ML accelerators, load/start models, NQ/DQ burst inferencing, and quantize/dequantize tensors. Marvell PMD implements Glow and TVM backends and a TVM runtime integration; test app shows ResNet50 at ~175 or 48 samples/sec and ResNet18 at ~376–865 samples/sec depending on quantization, with DPDK adding about 1.1 μs extra latency.
AI-written summary · about
chapters (7)
- 0:00Introduction to ML in DPDK
- 4:16ML in networking use cases
- 9:12ML device specification overview
- 15:38Data quantization and inference
- 18:31Current implementation status
- 20:40Performance observations
- 21:01Future roadmap
related talks
29mMulti-Accelerator PyTorch Serving With NxD Inference and vLLM
36mRunning High-performance User-space Packet Processing Apps in Kubernetes
36mBringing vDPA to Life in Kubernetes
43mLessons Learnt from Reusing QDMA NIC to Base Band PMD
51mAccelerate containers and VMs networking with vDPA
25mDPDK performance optimization for edge computing scenario
12mPreview: ML Serving
30mZXDH DPU Adapter and It's Application