Multi Channel DMA Intel® FPGA IP for PCI Express User Guide

ID 683821
Date 10/29/2021

A newer version of this document is available. Customers should click here to go to the newest version.

Document Table of Contents

8.3.5. Software Flow

Figure 44. Kernel Mode Driver Software Flow

Step 1

  • Application initializes the Libmcmem library & uses the library APIs to receive DMA memory
  • Libmcmem, as part of initialization allocates huge pages & sets them up into smaller chunks. It allocates these chunks to the application as per request.

Step 2

  • Application requests for channel through the character device. Kernel module driver receives the request and allocates an available channel. If all channels are busy, the request are not be served.
  • Application as part of initialization also provides the kernel driver with eventfds, payload size per descriptor & file size (optional) using a control message.
  • The eventfds are used by the driver to notify the application regarding the completion of DMA requests.
  • Kernel Driver initializes following registers which includes Starting address of descriptors, queue size, write back address for consumed Head, payload size in D2H descriptors as part of the channel initialization.
  • QCSR registers:
    • Q_START_ADDR_L (Offset 8’h08)
    • Q_START_ADDR_H (Offset 8’h0C)
    • Q_SIZE (Offset 8’h10)
    • Q_CONSUMED_HEAD_ADDR_L (Offset 8’h20)
    • Q_CONSUMED_HEAD_ADDR_H (Offset 8’h24)
    • Q_BATCH_DELAY (Offset 8’h28)
    • Q_CTRL (Offset 8’h00)
    • QDMA_REGS_2_Q_DEBUG_STATUS_4 (Q_PYLD_COUNT) (Offset 8'h44)
  • GCSR register:
    • WB_INTR_DELAY (Offset 8’h08)
  • It also sets up interrupt handlers for processing of MSIX signals, eventfds for forwarding completion events to the application
  • Application submits the DMA transfer requests to the driver through read/write APIs

Step 3

  • Hardware supports request descriptors for submission of requests. Kernel module manages a ring of descriptors for request submissions.
  • Each request received from the application is transformed into a descriptor & pushed into the descriptor ring
  • The descriptor contains the starting physical address of the DMA buffer allocated in Step 1
  • If channel is not busy and descriptor ring is not full it goes to step 3
  • If channel is busy or descriptors ring is full, the thread retries to initiate the transfer.

Step 4

  • The submission of the descriptor is done through updation of tail pointer FIFO of the FPGA logic
  • QCSR tail pointer register Q_TAIL_POINTER (Offset 8’h14) is updated by one.

Step 5

  • Once the tail pointer write happens, QDMA IP fetches the descriptors from host memory starting from the programmed Q_START_ADDR_L/H address.
  • QDMA IP parses the descriptor content to find the sources, destination addresses and length of the data from descriptor and starts DMA operation.

Step 6

  • Once descriptor processing is completed, IP notifies the completion status based on following methods.
  • Following are two different ways for completion status indication, which can be enabled in each descriptor.
    • Either based on MSI-X Interrupt : QDMA IP sends MSI-X interrupt to host if enabled in Q_CTRL.
    • Writeback: QDMA IP updates Q_CONSUMED_HEAD_ADDR_L/H, if writeback is enabled in Q_CTRL.

Step 7

  • In case of MSI-X Interrupt, interrupt handler of the kernel driver processes the signal and passes the count of completed requests to the application via eventfds
  • In case of Writeback, the application polls the character device to check on the completed requests. The driver passes the count of completed requests to the application via eventfds.