Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Guide

ID 683517
Date 7/07/2023
Public

A newer version of this document is available. Customers should click here to go to the newest version.

Document Table of Contents

3.5.2.3.1.3. Set the Boot Parameters

Follow the steps below to modify the default hugepages setting in the grub files:
  1. Edit the /etc/default/grub file

    Append the highlighted parameters to the GRUB_CMDLINE_LINUX line in the /etc/default/grub file

    CentOS: GRUB_CMDLINE_LINUX=" rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb default_hugepagesz=1G hugepagesz=1G hugepages=40 panic=1 intel_iommu=on iommu=pt

    Ubuntu: GRUB_CMDLINE_LINUX="default_hugepagesz=1G hugepagesz=1G hugepages=20 intel_iommu=on iommu=pt panic=1 quiet splash vt.handoff=7"

    File contents after the edit for CentOS is shown below:

    GRUB_TIMEOUT=5
    GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
    GRUB_DEFAULT=saved
    GRUB_DISABLE_SUBMENU=true
    GRUB_TERMINAL_OUTPUT="console"
    GRUB_CMDLINE_LINUX=" rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb default_hugepagesz=1G hugepagesz=1G hugepages=40 panic=1 intel_iommu=on iommu=pt
    GRUB_DISABLE_RECOVERY="true"

    In the case of memory allocation failure at the time of Virtual Function creation, add the following boot parameters: "pci=hpbussize=10,hpmemsize=2M,nocrs,realloc=on"

    To bind the device to vfio-pci and use IOMMU, enable the following parameter: intel_iommu=on

    To use UIO and not enable the IOMMU lookup, add the following parameter: iommu=pt

    To use the AMD platform and the UIO driver, add the following parameter at boot time: iommu=soft

    An example /etc/default/grub file on ubuntu after the edits can be seen below:
    root@bapvecise042:~# cat /etc/default/grub
    # If you change this file, run 'update-grub' afterwards to update 
    
    # /boot/grub/grub.cfg.
    # For full documentation of the options in this file, see: 
    
    #   info -f grub -n 'Simple configuration' 
    
    GRUB_DEFAULT="1>2" 
    
    GRUB_TIMEOUT_STYLE=hidden 
    
    GRUB_TIMEOUT=0 
    
    GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian` 
    
    GRUB_CMDLINE_LINUX_DEFAULT="quiet splash" 
    
    GRUB_CMDLINE_LINUX="default_hugepagesz=1G hugepagesz=1G hugepages=20 intel_iommu=on iommu=pt panic=1" 
    
    # Uncomment to enable BadRAM filtering, modify to suit your needs 
    
    # This works with Linux (no patch required) and with any kernel that obtains 
    
    # the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...) 
    
    #GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef" 
    
    # Uncomment to disable graphical terminal (grub-pc only) 
    
    #GRUB_TERMINAL=console 
      
    
    # The resolution used on graphical terminal 
    
    # note that you can use only modes which your graphic card supports via VBE 
    
    # you can see them in real GRUB with the command `vbeinfo' 
    
    #GRUB_GFXMODE=640x480 
    
    # Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux 
    
    #GRUB_DISABLE_LINUX_UUID=true 
      
    
    # Uncomment to disable generation of recovery mode menu entries 
    
    #GRUB_DISABLE_RECOVERY="true" 
      
    
    # Uncomment to get a beep at grub start 
    
    #GRUB_INIT_TUNE="480 440 1" 
  2. Generate GRUB configuration files.

    To check whether the boot system is legacy or EFI-based, check the existence of the following file:

    $ls -al /sys/firmware/efi

    If this file is present, the boot system is EFI-based. Otherwise, it is a legacy system.
    1. In case of a legacy system, execute the following command:

      $ grub2-mkconfig -o /boot/grub2/grub.cfg

    2. In case of an EFI-based system, execute the following command:

      $ grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg

    3. In case of Ubuntu, execute the following command:
      grub-mkconfig -o /boot/efi/EFI/ubuntu/grub.cfg
      or
      grub2-mkconfig -o /boot/efi/EFI/ubuntu/grub.cfg 
      
      or
      sudo grub update
  3. Reboot the system.
  4. Verify the changes above:

    $ cat /proc/cmdline

  5. Set the huge pages:

    $ echo 40 > /proc/sys/vm/nr_hugepages

  6. If host supports multiple NUMAs, follow the following steps:
    1. Check how many NUMAs enabled on host.
      $lscpu | grep NUMA
      NUMA node(s): 2
      NUMA node0 CPU(s): 0-15, 32-47
      NUMA node1 CPU(s): 16-31, 48-63

      In this example, we have 2 NUMAs. If only one NUMA is present, ignore this step:

    2. Check which device is provisioned:
      $cat /sys/class/pci_bus/<Domain:Bus>\
      /device/numa_node
    3. Enable the Huge pages, which ever NUMA, device is located:
      $echo 40> /sys/devices/system/node/node<nodenum>\
      /hugepages/hugepages-1048576kB/nr_hugepages
    4. Configure thread sequence based on which NUMA device is located, for example:
      #define THREAD_SEQ "0-15" in software/user/cli/\
      perfq_app/perfq_app.h