Characterizing NVIDIA Confidential Computing Overheads Across Model Inference & Training
summary
Measurement shows NVIDIA Confidential Computing overheads stem from DMA disabled between CPU and GPU, forcing a bounce buffer path that inflates CPU→GPU command and data crossings. Benchmarks on B300 and H200 with Intel TDX show decode overhead ~30% for CC‑naive VLM at low concurrency, falling to 6% and as low as ~1% with NVIDIA’s TRTLM optimizations. Kernel launches run 3–4× slower in latency (microseconds); NVLink encryption keeps bandwidth near Hopper. Use CUDA graphs, pageable (not pinned) memory, larger batches and fewer tensor‑parallel splits.
AI-written summary · about
chapters (18)
- 0:00Confidential computing overheads
- 1:03NVIDIA CC performance impact
- 2:26What is encrypted in CC
- 3:32CPU-GPU data movement under CC
- 4:35Inference phases and CPU-GPU comms
- 6:24Non-CC vs CC CPU-GPU comms
- 7:07Naive vs optimized CC inference
- 8:13Decode overhead with CC naive
- 9:36CC-aware optimizations impact on decode
- 10:39Training overheads under CC
- 11:22Tracing CC workload overheads
- 12:26Pin vs pageable memory under CC
- 13:11NVLink encryption impact on bandwidth
- 13:54Model parallelism under CC
- 14:37NVLink multicast limits in CC
- 15:19Strategies to reduce CC overheads
- 16:21Bounce buffer support in Linux kernel
- 17:04When CC optimizations are needed
related talks
47mTowards Confidential AI for the Masses!
36mCTO Panel: The Impact of Confidential Computing with Intel, Azure, AMD & NVIDIA
40mOptimize Your AI Cloud Infrastructure: A Hardware Perspective
19mOptimizing CNN Model Inference on CPUs
42mSponsored Session: Confidential Computing - New Capabilities for New Workloads
20mTrusted Hardware for AI Workloads: Extending Confidential...
29mBarraCUDA: GPUs Do Leak DNN Weights- 42mSponsored Session: Confidential Computing - New Capabilities for New Workloads