Quick Start Guide for Intel® Dynamic Load Balancer on 4th and 5th Gen Intel® Xeon® Scalable and Intel® Xeon® 6 Processors

Intel® DLB is a Peripheral Component Interconnect Express (PCIe) device that provides load-balanced, prioritized scheduling of events (packets) across CPU cores/threads, enabling efficient core-to-core communication. It is a hardware accelerator located inside the latest Intel® Xeon® CPUs offered by Intel. Under the hood, Intel® DLB is a hardware-managed system of queues and arbiters connecting producers and consumers. This article explores Intel® DLB and one exciting use case.

Intel DLB supports a Producer/Consumer model. A Producer is an agent that has a type of message to place into a queue. A Consumer is an agent that removes the message from the queue. These messages typically describe work for the consumer to execute.

Intel® DLB Linux Driver Build

DLB driver uses the kbuild build system. To build out-of-tree, simply run make. You can optionally provide the KSRC environment variable to specify the kernel source tree to build against. 

 $ cd dlb/driver/dlb2
 $ make

Intel® DPDK DLB PMD Build

The release package contains an add-on patch to the DPDK base package for DLB PMD support. DPDK base packages are available for download at www.dpdk.org. Be sure to download the version specified in the DPDK Version section of the Readme. Find more details on DLB PMD. 

To apply the add-on patch and build DPDK, follow these steps:

$ git clone https://github.com/DPDK/dpdk.git
$ cd dpdk
$ meson setup build
$ cd build
$ ninja
$ ninja install
$ ldconfig
$ patch -Np1 < $DLB_SRC_TOP/dlb/dpdk/<DPDK patch name>.patch
Ex: $ patch -Np1 < /home/administrator/ondemand/dlb/dpdk/dpdk_dlb_v21.11_e814edf7bf_diff.patch

$ mkdir -p /dev/hugepages
$ mountpoint -q /dev/hugepages || mount -t hugetlbfs nodev /dev/hugepages
$ echo 64 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages

Additional meson configuration and build options can be found on the DPDK site.
Find ports for binding to DPDK by running the tool with the -s option.

$ cd /home/administrator/DLB/dpdk-main/usertools/
./dpdk-devbind.py --status

Network devices using DPDK-compatible driver
============================================
<none>

Network devices using kernel driver
===================================
0000:29:00.0 '82571EB ... 10bc' if=enp41s0f0 drv=e1000e unused=
0000:29:00.1 '82571EB ... 10bc' if=enp41s0f1 drv=e1000e unused=
0000:2a:00.0 '82571EB ... 10bc' if=enp42s0f0 drv=e1000e unused=
0000:2a:00.1 '82571EB ... 10bc' if=enp42s0f1 drv=e1000e unused=

Other Network devices
=====================
<none>
....

You can find that network ports are bound to the kernel driver and not to DPDK. To bind a port to DPDK, run dpdk-devbind.py specifying a driver and a device ID. Device ID is a PCI address of the device or a more friendly style like eth0 found by ifconfig or ip command..

# Bind a port with 2a:00.0 (PCI address)
./usertools/dpdk-devbind.py --bind=uio_pci_generic 2a:00.0

# or eth0
./usertools/dpdk-devbind.py --bind=uio_pci_generic eth0

After binding two ports, you can find it is under the DPDK driver and cannot find it by using ifconfig or ip.

$ cd /home/administrator/DLB/dpdk-main/usertools
./dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:2a:00.0 '82571EB ... 10bc' drv=uio_pci_generic unused=vfio-pci
0000:2a:00.1 '82571EB ... 10bc' drv=uio_pci_generic unused=vfio-pci

Network devices using kernel driver
===================================
0000:29:00.0 '...' if=enp41s0f0 drv=e1000e unused=vfio-pci,uio_pci_generic
0000:29:00.1 '...' if=enp41s0f1 drv=e1000e unused=vfio-pci,uio_pci_generic

Other Network devices
=====================
<none>
....

Confirm DPDK is set up properly

For testing, you can confirm if you are ready to use DPDK by running DPDK’s sample application. 

$ cd /home/administrator/DLB/dpdk-main/examples/l2fwd
$ make

In this case, run this application simply with just two options, while DPDK has many kinds of options.

•    -l: core list
•    -p: port mask

$ cd /home/administrator/DLB/dpdk-main/examples/l2fwd/build
sudo ./l2fwd   -l 1-2   -- -p 0x1

It must be separated with -- to specify which option is for EAL or the application. Refer to the L2 Forwarding Sample Application for more details.

Tests for DPDK install

Check if DLB is active

$ lspci -d :2710
0000:78:00.0 Co-processor: Intel Corporation Device 2710
0000:f5:00.0 Co-processor: Intel Corporation Device 2710

DLB User Tools

$ cd /home/administrator/DLB/dpdk-main/usertools
./cpu_layout.py

./dpdk-devbind.py --status
 
./dpdk-hugepages.py --show

cd /home/administrator/DLB/dpdk-main/build/app
./dpdk-test / RTE>> ticketlock_autotest