Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Guide

Download PDF
ID 683517
Date 6/09/2025
Public

A newer version of this document is available. Customers should click here to go to the newest version.

Document Table of Contents
Document Table of Contents x
1. Introduction 2. Design Example Detailed Description 3. Design Example Quick Start Guide 4. Multi Channel DMA Intel FPGA IP for PCI Express Design Example User Guide Archives 5. Revision History for the Multi Channel DMA Intel FPGA IP for PCI Express Design Example User Guide
1. Introduction x
1.1. Terms and Acronyms 1.2. MCDMA IP Modes
2. Design Example Detailed Description x
2.1. Design Example Overview 2.2. PIO using MCDMA Bypass Mode 2.3. Avalon-ST Packet Generate/Check 2.4. Avalon-ST Device-side Packet Loopback 2.5. Avalon-MM DMA 2.6. BAM_BAS Traffic Generator and Checker 2.7. External Descriptor Controller
2.1. Design Example Overview x
2.1.1. H-Tile MCDMA IP - Design Examples for Endpoint 2.1.2. P-Tile MCDMA IP - Design Examples for Endpoint 2.1.3. F-Tile MCDMA IP - Design Examples for Endpoint 2.1.4. R-Tile MCDMA IP - Design Examples for Endpoint 2.1.5. Design Example BAR Mappings 2.1.6. Design Example Driver Support
2.2. PIO using MCDMA Bypass Mode x
2.2.1. Avalon-ST PIO using MCDMA Bypass Mode 2.2.2. Avalon-MM PIO using MCDMA Bypass Mode
2.2.1. Avalon-ST PIO using MCDMA Bypass Mode x
2.2.1.1. Single-Port Avalon-ST PIO using MCDMA Bypass Mode
2.2.2. Avalon-MM PIO using MCDMA Bypass Mode x
2.2.2.1. Simulation Results 2.2.2.2. Hardware Test Results
2.3. Avalon-ST Packet Generate/Check x
2.3.1. Single-Port Avalon-ST Packet Generate/Check
2.3.1. Single-Port Avalon-ST Packet Generate/Check x
2.3.1.1. Simulation Waveforms 2.3.1.2. Simulation Log 2.3.1.3. Hardware Test Result
2.4. Avalon-ST Device-side Packet Loopback x
2.4.1. Simulation Results 2.4.2. Hardware Test Results
2.5. Avalon-MM DMA x
2.5.1. Simulation Results 2.5.2. Hardware Test Results
2.6. BAM_BAS Traffic Generator and Checker x
2.6.1. BAM_BAS Traffic Generator and Checker Example Design Register Map
2.7. External Descriptor Controller x
2.7.1. Registers 2.7.2. Hardware Test Results
2.7.1. Registers x
2.7.1.1. GCSR Registers (Base address 64’h00000) 2.7.1.2. QCSR Registers(Base address 64’h10000)
3. Design Example Quick Start Guide x
3.1. Design Example Directory Structure 3.2. Generating the Example Design using Quartus® Prime 3.3. Simulating the Design Example 3.4. Compiling the Example Design in Quartus® Prime 3.5. Running the Design Example Application on a Hardware Setup
3.2. Generating the Example Design using Quartus® Prime x
3.2.1. Procedure
3.3. Simulating the Design Example x
3.3.1. Testbench Overview 3.3.2. Supported Simulators 3.3.3. Example Testbench Flow for DMA Test with Avalon-ST Packet Generate/Check Design Example 3.3.4. Run the Simulation Script 3.3.5. Steps to Run the Simulation 3.3.6. View the Results
3.3.5. Steps to Run the Simulation x
3.3.5.1. Steps to Run the Simulation : QuestaSim* / Questa* Intel® FPGA Edition 3.3.5.2. Steps to Run the Simulation : VCS* / VCS* MX 3.3.5.3. Steps to Run the Simulation : Xcelium*
3.5. Running the Design Example Application on a Hardware Setup x
3.5.1. Hardware Requirements 3.5.2. Software Requirements 3.5.3. Set Up the Hardware and Program the FPGA 3.5.4. Configuration Changes from BIOS 3.5.5. Host Operating System Check for Ubuntu v22.04 3.5.6. Installing the Required Kernel Version for Ubuntu v24.04 3.5.7. Set the Boot Parameters 3.5.8. MCDMA Custom Driver 3.5.9. MCDMA DPDK Poll Mode Driver 3.5.10. MCDMA Kernel Mode Network Device Driver
3.5.8. MCDMA Custom Driver x
3.5.8.1. Prerequisites 3.5.8.2. PIO Test 3.5.8.3. DMA Test 3.5.8.4. BAM Test 3.5.8.5. BAS Test
3.5.8.1. Prerequisites x
3.5.8.1.1. Install the Linux Kernel Driver 3.5.8.1.2. Enable Virtual Function 3.5.8.1.3. Build and Install the User Space Library 3.5.8.1.4. Build the Reference Application
3.5.8.2. PIO Test x
3.5.8.2.1. Custom PIO Read Write Test
3.5.8.3. DMA Test x
3.5.8.3.1. Avalon-MM DMA (AVMM DMA) 3.5.8.3.2. Avalon-ST Device-side Packet Loopback (Device-side Packet Loopback) 3.5.8.3.3. Avalon-ST Packet Generate/Check (Packet Generate/Check) 3.5.8.3.4. External Descriptor Controller
3.5.8.3.2. Avalon-ST Device-side Packet Loopback (Device-side Packet Loopback) x
3.5.8.3.2.1. DMA Test Beyond 256 Channels
3.5.8.3.3. Avalon-ST Packet Generate/Check (Packet Generate/Check) x
3.5.8.3.3.1. DMA Test Beyond 256 Channels
3.5.9. MCDMA DPDK Poll Mode Driver x
3.5.9.1. Prerequisites 3.5.9.2. PIO Test 3.5.9.3. DMA Test 3.5.9.4. BAM Test 3.5.9.5. BAS Test
3.5.9.1. Prerequisites x
3.5.9.1.1. Install the PMD Driver 3.5.9.1.2. Bind the Device 3.5.9.1.3. Enable Virtual Function
3.5.9.3. DMA Test x
3.5.9.3.1. Avalon-MM DMA (AVMM DMA) 3.5.9.3.2. Avalon-ST Packet Generate/Check (Packet Generate/Check) 3.5.9.3.3. Avalon-ST Device-side Packet Loopback (Device-side Packet Loopback)
3.5.9.3.2. Avalon-ST Packet Generate/Check (Packet Generate/Check) x
3.5.9.3.2.1. DMA Test Beyond 256 Channels
3.5.9.3.3. Avalon-ST Device-side Packet Loopback (Device-side Packet Loopback) x
3.5.9.3.3.1. DMA Test Beyond 256 Channels
3.5.10. MCDMA Kernel Mode Network Device Driver x
3.5.10.1. Prerequisites 3.5.10.2. PIO Test 3.5.10.3. Ping Test
3.5.10.1. Prerequisites x
3.5.10.1.1. Build and Install Netdev Driver 3.5.10.1.2. Enable Virtual Function
1. Introduction
2. Design Example Detailed Description
3. Design Example Quick Start Guide
4. Multi Channel DMA Intel FPGA IP for PCI Express Design Example User Guide Archives
5. Revision History for the Multi Channel DMA Intel FPGA IP for PCI Express Design Example User Guide

3.5.7. Set the Boot Parameters

Follow the steps below to modify the default hugepages setting in the grub files:
  1. Edit the /etc/default/grub file.

    Append the highlighted parameters to the GRUB_CMDLINE_LINUX line in the /etc/default/grub file.

    For Intel CPU:

    GRUB_CMDLINE_LINUX="default_hugepagesz=1G hugepagesz=1G hugepages=20 intel_iommu=on iommu=pt processor.max_cstate=0 intel_idle.max_cstate=0 intel_pstate=disable panic=1 quiet splash vt.handoff=7"

    For AMD CPU:

    GRUB_CMDLINE_LINUX="default_hugepagesz=1G hugepagesz=1G hugepages=20 amd_iommu=on iommu=soft processor.max_cstate=0 amd-pstate panic=1 quiet splash vt.handoff=7"

    An Intel CPU example of the /etc/default/grub file on Ubuntu after the edits can be seen below: 

    root@bapvecise042:~# cat /etc/default/grub # If you change this file, run 'update-grub' afterwards to update 
# /boot/grub/grub.cfg. 
# For full documentation of the options in this file, see: 
#   info -f grub -n 'Simple configuration' 
GRUB_DEFAULT="1>2" GRUB_TIMEOUT_STYLE=hidden 
GRUB_TIMEOUT=0 GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian` 
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash" 
GRUB_CMDLINE_LINUX="default_hugepagesz=1G hugepagesz=1G hugepages=20 intel_iommu=on iommu=pt nprocessor.max_cstate=0 
intel_idle.max_cstate=0 intel_pstate=disable panic=1 quiet splash vt.handoff=7"
# Uncomment to enable BadRAM filtering, modify to suit your needs 
# This works with Linux (no patch required) and with any kernel that obtains 
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...) 
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef" 
# Uncomment to disable graphical terminal (grub-pc only) 
#GRUB_TERMINAL=console 
# The resolution used on graphical terminal 
# note that you can use only modes which your graphic card supports via VBE 
# you can see them in real GRUB with the command `vbeinfo' 
#GRUB_GFXMODE=640x480 
# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux 
#GRUB_DISABLE_LINUX_UUID=true # Uncomment to disable generation of recovery mode menu entries 
#GRUB_DISABLE_RECOVERY="true" # Uncomment to get a beep at grub start 
#GRUB_INIT_TUNE="480 440 1"
  2. Generate GRUB configuration files.

    Execute the following command:

    sudo update-grub

  3. Reboot the system.
  4. Verify the changes above by running the following command:

    $ cat /proc/cmdline

  5. Set the hugepages:

    $ echo 40 | sudo tee /proc/sys/vm/nr_hugepages

  6. If the host supports multiple NUMAs, use the following steps:
    1. Check how many NUMAs are enabled on the host.
      $lscpu | grep NUMA
NUMA node(s): 2
NUMA node0 CPU(s): 0-15, 32-47
NUMA node1 CPU(s): 16-31, 48-63

      In this example, we have two NUMAs. If only one NUMA is present, ignore the below steps:

    2. Check which device is provisioned:
      $cat /sys/class/pci_bus/<Domain:Bus>/device/numa_node
    3. Enable the hugepages, whichever NUMA device is located:
      $echo 40 |> sudo tee /sys/devices/system/node/node<nodenum>\
/hugepages/hugepages-1048576kB/nr_hugepages
    4. Configure the thread sequence in software/user/cli/perfq_app/perfq_app.h based on which NUMA device is located. For example:

      #define THREAD_SEQ "0-15"

  7. Set PCIe_SLOT based on your design example link width configuration. Modify the software flag below in the p0_software/user/common/include/ifc_libmqdma.h and p0_software/dpdk/dpdk/drivers/net/mcdma/rte_pmd_mcdma.h files.

    #define PCIe_SLOT 0 /* 0 - x16, 1 - x8, 2 – x4 */

    Example of software flag setting for x4 link width:

    #define PCIe_SLOT 2

Level Two Title