Configuring iWARP RDMA for Intel Ethernet in Microsoft Windows*

Documentation

Install & Setup

000031906

02/14/2020

Remote Direct Memory Access, or RDMA, allows a computer to access another computer's memory without interacting with either computer's operating system data buffers. Therefore, networking speed and throughput are increased. iWARP is a protocol for implementing RDMA across Internet Protocol networks.

Microsoft* Windows* provides two forms of RDMA: Network Direct Kernel (NDK) and Network Direct (ND).
 

iWARP RDMA for NDK Features such as SMB Direct (Server Message Block)

NDK allows Windows components (such as SMB Direct storage) to use iWARP features.

iWARP NDK Configuration

NDK functionality is included in the Intel base networking drivers and requires no additional features to be installed.

How do I verify iWARP RDMA operation?

Check that RDMA is enabled on the network interfaces.

Get-NetAdapterRDMA

Verify that the network interfaces show as RDMA capable and multichannel is enabled.

Get-SmbClientNetworkInterface

Network Direct is enabled in the OS.

Get-NetOffloadGlobalSetting | Select NetworkDirect

Each RDMA-capable network interface should have a listener at port 445 (Windows Client OSs that support RDMA may not post listeners).

netstat.exe -xan | ? {$_ -match "445"}
 

How do I enable iWARP routing across IP Subnets?

If you want to allow NDK's RDMA feature across subnets, you'll need to select Enable iWARP routing across IP Subnets on the iWARP Configuration Options screen during base driver installation (See the iWARP User Mode Installation section below).

How do I use iWARP for virtualized environments?

In order to enable RDMA capability on virtual adapter(s) connected to a VMSwitch, *SRIOV (Single Root IO Virtualization) and *VMQ (Virtual Machine Queues) advanced properties must be enabled on each port driver. Under certain circumstances, these settings may be disabled by default. You can set these options manually in the advanced tab of the adapter properties dialog box, or the following PowerShell commands may be used:

  • Set-NetAdapterAdvancedProperty -Name <nic_name> -RegistryKeyword *SRIOV -RegistryValue 1
  • Set-NetAdapterAdvancedProperty -Name <nic_name> -RegistryKeyword *VMQ -RegistryValue 1

 

How do I test iWARP NDK: Microsoft* Windows* SMB Direct with DiskSPD?

This section outlines a recommended way to test iWARP RDMA for Intel Ethernet functionality and performance on Microsoft* Windows* operating systems.

Note that since SMB Direct is a storage workload, the performance of the benchmark may be limited to the speed of the storage device rather than the network interface being tested. Intel recommends using the fastest storage possible in order to test the true capabilities of the network device(s) under test.

Test instructions:

  1. Set up at least two Microsoft* Windows* Server 2012 R2 or Microsoft* Windows* Server 2016 servers with at least one network interface of iWARP-capable Intel® Ethernet connected per server.
  2. On the system designated as the SMB server, set up an SMB share. Note that the performance of the benchmark may be limited to the speed of the storage device rather than the network interface being tested. Storage setup is outside of the scope of this document.
    PowerShell (PS) command:
    New-SmbShare -Name <SMBsharename> -Path <SMBsharefilepath> -FullAccess <domainname>\Administrator,Everyone


    Example:
    New-SmbShare -Name RAMDISKShare -Path R:\RAMDISK -FullAccess group\Administrator,Everyone

     
  3. Download and install Microsoft utility Diskspd
  4. Using CMD or PowerShell, cd to the DiskSpd folder and run tests (Refer to Diskspd documentation for more details on parameters).

    Example: Set the block size to 4K, run the test for 60 seconds, disable all hardware and software caching, measure and display latency statistics, leverage 16 overlapped IOs and 16 threads per target, random 0% writes and 100% reads and create a 10-GB test file at
    “\\<SMBserverX722IP>\<SMBsharename>\test.dat” :
    .\diskspd.exe –b4K -d60 -h -L –o16 -t16 –r –w0 -c10G \\<SMBserverX722IP>\<SMBsharename>\test.dat

     
  5. Verify that RDMA traffic is running using perfmon counters such as RDMA Activity and SMB Direct Connection. Refer to Microsoft documentation for more details.

How do I configure iWARP Guest Support (NDK Mode 3)?

NDK Mode 3 allows kernel mode Windows components to use iWARP features inside Hyper-V guest partitions. To enable NDK mode 3 on an Intel Ethernet device, do the following:

  1. Enable SR-IOV in your system's BIOS or uEFI.
  2. Enable the SR-IOV advanced option on the device.
  3. Configure the device with SR-IOV enabled on the switch. Perform this step for all functions on the same device:
    New-VMSwitch -Name <switch_name> -NetAdapterName <device_name>
    -EnableIov $true
  4. Configure the number of RDMA virtual functions (VFs) on the device by setting the RdmaMaxVfsEnabled advanced option. You must set all physical functions to the same value. The value is the maximum number of VFs that can be capable of RDMA at one time for the entire device. Enabling more VFs will restrict RDMA resources from physicals functions (PFs) and other VFs.
    Set-NetAdapterAdvancedProperty -Name <device_name> -RegistryKeyword RdmaMaxVfsEnabled -RegistryValue <Value: 0 – 32>
  5. Disable all PF adapters on the host and re-enable them. This is required when the registry keyword RdmaMaxVfsEnabled is changed or when creating or destroying a VMSwitch.
    Get-NetAdapterRdma | Disable-NetAdapter
     
    Get-NetAdapterRdma | Enable-NetAdapter
  6. Create VM Network Adapters for VMs that require RDMA VF support.
    Add-VMNetworkAdapter -VMName <vm_name> -VMNetworkAdapterName <device_name> -SwitchName <switch_name>
  7. If you plan to use Microsoft Windows 10 Creators Update (RS2) or later on a guest partition, set the RDMA weight on the VM Network Adapter by entering the following command on the host:
    Set-VMNetworkAdapterRdma -VMName <vm_name> -VMNetworkAdapterName <device_name> -RdmaWeight 100
  8. Set SR-IOV weight on the VM Network Adapter (Note: You must set SR-IOV weight to 0 before setting the RdmaWeight to 0):
    Set-VMNetworkAdapter -VMName <vm_name> -VMNetworkAdapterName <device_name> -IovWeight 100
  9. Install the VF network adapter with the PROSET Installer in the VM.
  10. Enable RDMA on the VF driver and Hyper-V Network Adapter using PowerShell in the VM:
    Set-NetAdapterAdvancedProperty -Name <device_name> -RegistryKeyword RdmaVfEnabled -RegistryValue 1
     
    Get-NetAdapterRdma | Enable-NetAdapterRdma

iWARP RDMA for ND User-Mode Applications

ND allows user-mode applications to use iWARP features. If you plan to make use of iWARP features in user-mode applications you're developing, you'll need to install the Network Direct (ND) feature when you install the drivers.

iWARP User Mode Installation

The Intel® Ethernet User Mode iWARP Provider is supported on Microsoft* Windows Server* 2012 R2 or later.

Note User mode applications may have prerequisites such as Microsoft HPC Pack or Intel MPI Library. Refer to your application documentation for more details.

Follow the steps below to install user-mode Network Direct (ND) iWARP features.

  1. From the installation media, run Autorun.exe to launch the installer. Choose Install Drivers and Software and accept the license agreement.
  2. On the Setup Options screen, select Intel® Ethernet User Mode iWARP Provider.
  3. On the iWARP Configuration Options screen, select Enable iWARP routing across IP Subnets if desired. Note that this option is displayed during base driver installation even if user mode iWARP wasn't selected. This option is applicable to Network Direct Kernel functionality as well.
  4. If Windows Firewall is installed and active, select Create an Intel® Ethernet iWARP Port Mapping Service rule in Windows Firewall and the networks to which to apply the rule. If Windows Firewall is disabled or you're using a third party firewall, you'll need to manually add this rule.
  5. Continue with driver and software installation.
Related topics
Which Intel® Ethernet Network Adapters Support iWARP?
iWARP* RDMA Here and Now Technology Brief
Leveraging RDMA Technologies to Accelerate Ceph* Storage Solutions
iWarp: The Movie