Multi Channel DMA Intel® FPGA IP for PCI Express* User Guide

ID 683821
Date 7/30/2024
Public
Document Table of Contents

8.1.1. Architecture

The figure below shows the software architecture block diagram of MCDMA custom driver.

Figure 47. Block Level Software Architecture

In the above block diagram, dotted lines represent memory mapped I/O interface. The other two lines represent read and write operations triggered by the device.

The Multi Channel DMA IP for PCI Express supports the following kernel based modules to expose the device to user space.
  • vfio-pci
  • UIO

These drivers do not perform any device management and indicate to the Operating System (OS) that the devices are being used by user space such that the OS does not perform any action (e.g. scanning the device etc.) on these devices.

vfio-pci

This is the secure kernel module, provided by kernel distribution. This module allows you to program the I/O Memory Managnment Unit (IOMMU). IOMMU is the hardware which helps to ensure memory safety in user space drivers. In case, if you are using Single Root I/O Virtualization (SR-IOV) , you can load vfio-pci and bind the device.

  • This module enables IOMMU programming and Function level reset (FLR)
  • To expose device Base Address Registers (BAR) to user space, vfio-pci enables ioctl
  • Supports MSI-X (Message Signal Interrupts extensions) interrupts
  • Kernel versions >= 5.7, supports the enablement of virtual functions by using sysfs interface.
If you are using kernel versions below 5.7, you have the following alternatives:
  • Use ifc_uio, which supports to enable VFs.
  • Apply the patch on kernel to enable virtual functions by using sysfs. It needs kernel rebuild.

ifc_uio

This is the alternative driver to vfio-pci, which doesn’t use IOMMU.

By using PCIe, sysfs, interrupt framework utilities this module reads allows the user space to access the device.

Like vfio-pci, this module can also be used from guest VM through the hypervisor. This driver allows the enablement/disablement of virtual functions. Once a virtual function is created, by default it binds to ifc_uio. Based on the requirement, you may unbind and bind to another driver.

Following are the functionalities supported by using this module:
  • Allows enablement/disablement of virtual functions by using sysfs interface.
  • Probes and exports channel BARs to libmqdma
  • Supports Interrupt notification/clearing

libmqdma

This is a user-space library used by the application to access the PCIe device.
  • This library has the APIs to access the MCDMA IP design and you can develop your application using this API.
  • It features calls for allocation, release and reset of the channels
  • libmqdma supports accessing the devices binded by two user space drivers UIO (uio) or Virtual Function I/O (VFIO) (vfio-pci).

You can tune these options from the make file.

In case of UIO, ifc_uio driver reads the BAR register info by using sysfs and register MSI-X info by using eventfds.

In case of VFIO, user space uses IOCTL command to read BAR registers, MSIX information and programming of IOMMU table.

Typically, when an application is running in a virtualized environment, you bind the device to vfio-pci module and libmqdma can access the device by using ioctl . Currently, the support of UIO and VFIO can be switched from common.mk file. UIO is enabled by default.

Sample application

This application uses the APIs from libmqdma and takes the following command line arguments as the input.
  • Total message sizes/ time duration
  • Packet size per descriptor
  • Write/Read
  • Completion reporting method
  • Number of channels

It runs multiple threads for accessing the DMA channel. It also has performance measuring capabilities. Based on the number threads you are using and number of channels you are processing, queues are scheduled on threads.