Intel® Acceleration Stack User Guide: Intel FPGA Programmable Acceleration Card N3000

Download
ID 683040
Date 6/14/2021
Public
Document Table of Contents

9. Testing Network Loopback Using Data Plane Development Kit (DPDK)

Before starting Data Plane Development Kit (DPDK), you must perform configuration steps described in Configuring Ethernet Interfaces as it relies on the FPGA being bound to OPAE driver (pac_n3000_net). While using DPDK, FPGA is unbound from this driver and bound to the vfio-pci driver.

Follow these steps to install DPDK for testing network loopback:
  1. You must enable the Intel IOMMU driver on the host. Complete the following steps to enable the Intel IOMMU driver:
    1. Add iommu=pt intel_iommu=on to the GRUB_CMDLINE_LINUX entry by editing /etc/default/grub. For example:
      GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/\
      swap rhgb quiet pci=realloc intel_iommu=pt"
      

      For RHEL: Additionally, add pci=realloc to GRUB_CMDLINE_LINUX entry.

    2. GRUB reads its configuration from either the /boot/grub2/grub.cfg file on traditional BIOS-based machines or from the /boot/efi/EFI/redhat/ grub.cfg file on UEFI machines. Depending on your system, execute one of the instructions below as root:
      • BIOS based machine:
        grub2-mkconfig -o /boot/grub2/grub.cfg
      • UEFI based machine:
        grub2-mkconfig -o /boot/efi/EFI/redhat/ grub.cfg
        # grub2-mkconfig -o /boot/efi/EFI/redhat/
        Generating grub configuration file ...
        Found linux image: /boot/vmlinuz-3.10.0-957.el7.x86_64
        Found initrd image: /boot/initramfs-3.10.0-957.el7.x86_64.img
        Found linux image: /boot/vmlinuz-0-rescue-594cabaaf9a84c6ea0a5167c89ad916d
        Found initrd image: /boot/initramfs-0-rescue-594cabaaf9a84c6ea0a5167c89ad916d.img
        /usr/sbin/grub2-mkconfig: line 290: /boot/efi/EFI/redhat/: Is a directory
        
    3. Reboot the server to apply the new GRUB configuration file.
    4. To verify the GRUB update, run the following command:
      $ cat /proc/cmdline
      The sample output below shows intel_iommu=on on the kernel command line.
      BOOT_IMAGE=/vmlinuz-3.10.0-957.el7.x86_64 root=/dev/mapper/rhel-root ro default_hugepagesz=1G hugepagesz=1G hugepages=64 hugepagesz=2M hugepages=2048 nosoftlockup mce=ignore_ce audit=0 isolcpus=1-11,24-35,13-23,36-47 nohz_full=1-11,24-35,13-23,36-47 rcu_nocbs=1-11,24-35,13-23,36-47 pci=realloc intel_iommu=on iommu=pt enforcing=0 crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet skew_tick=1
  2. Install the required packages:
    $	sudo yum install readline-devel libpcap libpcap-devel numactl-devel
    You need to download these two extra Fedora packages: To install:
    $ sudo rpm -i <RPM file>
    To check installation:
    $ rpm -qa | grep libfdt
  3. Compile and bind drivers:
    1. Download DPDK code from DPDK community and checkout release 19.08.
      $ git clone https://github.com/DPDK/dpdk.git
      $ cd dpdk
      $ git pull
      Already up-to-date.
      $ git reset --hard 31b798a6f08e9b333b94b8bb26910209aa810b73
      HEAD is now at 31b798a build: avoid overlinking
      $ wget https://patches.dpdk.org/series/6821/mbox/
      Length: 179795 (176K) [text/plain]
      Saving to: \u2018index.html\u2019
      
      100%[==============>] 179,795      554KB/s   in 0.3s   
      
      (554 KB/s) - \u2018index.html\u2019 saved [179795/179795]
      $ git am index.html
      Applying: net/i40e: i40e support ipn3ke FPGA port bonding
      Applying: raw/ifpga/base: add irq support
      Applying: raw/ifpga/base: clear pending bit
      Applying: raw/ifpga/base: add SEU error support
      Applying: raw/ifpga/base: add device tree support
      Applying: raw/ifpga/base: align the send buffer for SPI
      Applying: raw/ifpga/base: add sensor support
      Applying: raw/ifpga/base: introducing sensor APIs
      Applying: raw/ifpga/base: update SEU register definition
      Applying: raw/ifpga: add SEU error handler
      Applying: raw/ifpga: add PCIe BDF devices tree scan
      Applying: net/ipn3ke: remove configuration for i40e port bonding
      Applying: raw/ifpga/base: add secure support
      Applying: raw/ifpga/base: configure FEC mode
      Applying: raw/ifpga/base: clean fme errors
      Applying: raw/ifpga/base: add new API get board info
      Applying: raw/ifpga: add lightweight fpga image support
      Applying: raw/ifpga/base: add multiple cards support
      
      $ vim config/common_linux
      // modify CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV=n to CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV=y
      $ vim config/common_base
      // modify CONFIG_RTE_LIBRTE_IPN3KE_PMD=n to CONFIG_RTE_LIBRTE_IPN3KE_PMD=y
    2. Build DPDK and export RTE_SDK path to point to dpdk directory:
      Note: Ignore the message “Build complete [x86_64-native-linuxapp-gcc] Installation cannot run with T defined and DESTDIR undefined"
      $	export RTE_SDK=$PWD
      $	export RTE_TARGET=x86_64-native-linuxapp-gcc
      $	make config T=x86_64-native-linuxapp-gcc
      $	make install -j8 T=x86_64-native-linuxapp-gcc
      Note: $RTE_SDK points to the extracted dpdk source location.
    3. Bind FPGA and NIC to vfio-pci driver as shown below. The sample output below shows result for 2x2x25G configuration. The 8x10G configuration produces similar output with following exception:
      Table 11.  Output Differences
      Configuration Ports per Intel Ethernet Controller XL710-BM2 NIC Device ID for Intel Ethernet Controller XL710-BM2 NIC Ports
      2x2x25G 2 0d58
      8x10G 4 0cf8
      $	cd $RTE_SDK
      Check the binding between driver and device with the following command:
      $	./usertools/dpdk-devbind.py --status-dev net
      Sample output:
      Network devices using DPDK-compatible driver
      ============================================
      <none>
      Network devices using kernel driver
      ===================================
      0000:18:00.0 'NetXtreme BCM5720 Gigabit Ethernet PCIe 165f' if=em1 drv=tg3 unused= 
      0000:18:00.1 'NetXtreme BCM5720 Gigabit Ethernet PCIe 165f' if=em2 drv=tg3 unused= *Active*
      0000:19:00.0 'NetXtreme BCM5720 Gigabit Ethernet PCIe 165f' if=em3 drv=tg3 unused= 
      0000:19:00.1 'NetXtreme BCM5720 Gigabit Ethernet PCIe 165f' if=em4 drv=tg3 unused= 
      0000:14:00.0 'Device 0d58' if=p1p1 drv=i40e unused= 
      0000:14:00.1 'Device 0d58' if=p1p2 drv=i40e unused= 
      0000:16:00.0 'Device 0d58' if=p1p3 drv=i40e unused= 
      0000:16:00.1 'Device 0d58' if=p1p4 drv=i40e unused= 
      Other Network devices
      =====================
      0000:15:00.0 'Device 0b30' unused=intel_fpga_pci
      Install the vfio-pci kernel driver:
      $	sudo modprobe vfio-pci
      Bind the FPGA and FVL PFs to DPDK driver. Replace Bus:Device.Function with your output from above step. In case of 8x10G image, bind all 8 FVL B:D.F and the FPGA to vfio_pci:
      $	sudo ./usertools/dpdk-devbind.py -b vfio-pci 14:00.0 \
      14:00.1 15:00.0 16:00.0 16:00.1
      Rerun ./usertools/dpdk-devbind.py –-status-dev net to check that FPGA and FVL PF’s are bound to vfio-pci driver.
      Figure 17. Sample Output
    4. Reserve hugepages:
      $ sudo mkdir -p /mnt/huge
      $ sudo mount -t hugetlbfs nodev /mnt/huge
      $ sudo sh -c "echo 2048 > /sys/kernel/mm/hugepages/\
      hugepages-2048kB/nr_hugepages"

For more information on how the FPGA support is enabled in DPDK, refer to Data Plane Development Kit Reference Manual: Intel FPGA Programmable Acceleration Card N3000 .

Did you find the information on this page useful?

Characters remaining:

Feedback Message