The figure below shows the software architecture block diagram of MCDMA custom driver.
In the above block diagram, dotted lines represent memory mapped I/O interface. The other two lines represent read and write operations triggered by the device.
These drivers do not perform any device management and indicate to the Operating System (OS) that the devices are being used by user space such that the OS does not perform any action (e.g. scanning the device etc.) on these devices.
This is the secure kernel module, provided by kernel distribution. This module allows you to program the I/O Memory Managnment Unit (IOMMU). IOMMU is the hardware which helps to ensure memory safety in user space drivers. In case, if you are using Single Root I/O Virtualization (SR-IOV) , you can load vfio-pci and bind the device.
- This module enables IOMMU programming and Function level reset (FLR)
- To expose device Base Address Registers (BAR) to user space, vfio-pci enables ioctl
- Supports MSI-X (Message Signal Interrupts extensions) interrupts
- Kernel versions >= 5.7, supports the enablement of virtual functions by using sysfs interface.
- Use ifc_uio, which supports to enable VFs.
- Apply the patch on kernel to enable virtual functions by using sysfs. It needs kernel rebuild.
This is the alternative driver to vfio-pci, which doesn’t use IOMMU.
By using PCIe, sysfs, interrupt framework utilities this module reads allows the user space to access the device.
Like vfio-pci, this module can also be used from guest VM through the hypervisor. This driver allows the enablement/disablement of virtual functions. Once a virtual function is created, by default it binds to ifc_uio. Based on the requirement, you may unbind and bind to another driver.
- Allows enablement/disablement of virtual functions by using sysfs interface.
- Probes and exports channel BARs to libmqdma
- Supports Interrupt notification/clearing
- This library has the APIs to access the MCDMA IP design and you can develop your application using this API.
- It features calls for allocation, release and reset of the channels
- libmqdma supports accessing the devices binded by UIO or Virtual Function I/O (VFIO ).
You can tune these options from the make file.
In case of UIO, ifc_uio driver reads the BAR register info by using sysfs and register MSI-X info by using eventfds.
In case of VFIO, user space uses IOCTL command to read BAR registers, MSIX information and programming of IOMMU table.
Typically, when an application is running in a virtualized environment, you bind the device to vfio-pci module and libmqdma can access the device by using ioctl . Currently, the support of UIO and VFIO can be switched from common.mk file. UIO is enabled by default.
- Total message sizes/ time duration
- Packet size per descriptor
- Completion reporting method
- Number of channels
It runs multiple threads for accessing the DMA channel. It also has performance measuring capabilities. Based on the number threads you are using and number of channels you are processing, queues are scheduled on threads.