How to capture 100G Ethernet traffic at wire speed to local disk
summary
Shows how to capture 100G (12.5 GB/s) into a single 1U box by combining Mellanox ConnectX‑6 NICs and userland/driver techniques (DPDK/SPDK or RDMA verbs) or a heavily tuned kernel+pagecache path, with 32 NVMe devices (Intel Ruler style) delivering aggregated ~40–56 GB/s. Key bottlenecks are 12.5 GB/s per 100G, 50M pps for small packets, 4KiB page‑cache overhead, PCIe/NUMA placement and contiguous memory fragmentation; remedies are per‑socket filesystems, CPU/thread pinning, BIOS sub‑NUMA tuning, and occasional reboots to regain contiguous memory.
AI-written summary · about
chapters (10)
- 0:00How to capture 100G Ethernet traffic at wire speed to local disk
- 2:17High Volume Data Capture results in Two Problems
- 4:26Approaches to receive 100G Traffic
- 8:26Special 100G Hardware Required? The only known NCs that support 100G with bulking are the Melanox NICS
- 11:10Page cache performance issues
- 16:22The Storage Architecture 2 Sockets with 384 Gbyte of memory each and a 20 Core processor
- 19:14Sub NUMA Clustering Setup with 4 Nodes
- 20:22BIOS issues with SNC SNC spits the Skylake processor into two halves each with their own memory controller and PCle links
- 21:12Tests were done using the "fio" tool
- 22:39Single Thread Performance
related talks
47mThe World of 100G Networking
34mCan the Linux networking stack be used with very high speed applications?
16mA High-Speed Stateful Packet Processing Approach for Tbps Programmable Switches
14mHigh-throughput and Flexible Host Networking for Accelerated Computing
31mA Practical, Research‑Backed Introduction To Cloud‑Native Kernel‑Bypass Networking
20mNetKernel: Making Network Stack Part of the Virtualized Infrastructure
26mApplying New High-Throughput Network Solutions in IEC (Integrated Edge Cloud)
02h 53mPreview Sessions