iWARP Throughput Performance Issue on Intel® Ethernet Network Adapter E810-CQDA2

• Throughput is only around 49Gb/s when InfinBand send bandwidth (ib_send_bw) with only one Queue Pair (QP).
• With two QPs, this bumps up to around 88 Gb/s and with three up to 91 Gb/s

Intel® Ethernet Network Adapter E810-CQDA2 is designed to deliver best performance using multiple Queue Pairs (QP) hence, it cannot reach maximum bandwidth with a single QP. This is the case whether the protocol is set to Internet Wide-area RDMA Protocol (iWARP) or RDMA over Converged Ethernet (RoCEv2) mode on E810s.

This is documented in the README_irdma.txt file within the Remote Direct Memory Access (iRDMA) Driver

Performance
RDMA performance may be optimized by adjusting system, application, or driver settings.

• Flow control is required for best performance in RoCEv2 mode and is optional in iWARP mode. Both link-level flow control (LFC) and priority flow control (PFC) are supported, but PFC is recommended. See the Flow Control Settings section of this document for configuration details.
• For bandwidth applications, multiple queue pairs (QPs) are required for best performance. For example, in the perftest suite, use -q 8 on the command line to run with 8 QP.