GTS AXI Multichannel DMA IP for PCI Express User Guide

Download PDF
ID 847470
Date 5/06/2025
Public

A newer version of this document is available. Customers should click here to go to the newest version.

Document Table of Contents
Document Table of Contents x
1. Overview 2. Quick Start Guide 3. Configuring and Generating the GTS AXI Multichannel DMA IP for PCI Express 4. Integrating the IP With Your Application 5. Simulating the IP 6. Validating the IP A. Appendix A: Functional Description B. Appendix B: Registers C. Document Revision History for the GTS AXI Multichannel DMA IP for PCI Express*
1. Overview x
1.1. About the GTS AXI Multichannel DMA IP for PCI Express* 1.2. About the Document 1.3. Applications 1.4. Key Concepts 1.5. IP Design Flow 1.6. Design Guidelines 1.7. Design Flow Requirements
1.1. About the GTS AXI Multichannel DMA IP for PCI Express* x
1.1.1. IP Release Information 1.1.2. IP Features 1.1.3. IP High-level Block Diagram 1.1.4. Licensing the IP 1.1.5. Device Family Support 1.1.6. Device Speed Grade Support 1.1.7. Resource Utilization 1.1.8. IP and Design Example Support
1.1.2. IP Features x
1.1.2.1. Root Port Mode Features 1.1.2.2. Endpoint Mode Features
1.2. About the Document x
1.2.1. Target Audience 1.2.2. Conventions Used in the Document 1.2.3. Terminology 1.2.4. Acronyms 1.2.5. Reference Documents and Training
1.4. Key Concepts x
1.4.1. PCI Express and DMA 1.4.2. Altera FPGA IPs for PCI Express and DMA Controller 1.4.3. Example Use Case
1.6. Design Guidelines x
1.6.1. Supporting IPs 1.6.2. Aligning IP Settings with the GTS AXI Streaming IP for PCI Express
1.7. Design Flow Requirements x
1.7.1. Software Requirements 1.7.2. Hardware Requirements 1.7.3. Supported EDA Tools
2. Quick Start Guide x
2.1. Step-by-step IP Bring-up 2.2. Reset Sequence
2.1. Step-by-step IP Bring-up x
2.1.1. Downloading and Installing Quartus® Prime Software 2.1.2. Configuring and Generating the GTS AXI Multichannel DMA IP for PCI Express 2.1.3. Configuring and Generating the GTS AXI Streaming Intel FPGA IP for PCI Express 2.1.4. Configuring and Generating the GTS System PLL Clocks Intel FPGA IP 2.1.5. Configuring and Generating the GTS Reset Sequencer Intel FPGA IP 2.1.6. Configuring and Generating the Reset Release IP 2.1.7. Instantiating and Connecting the IP Interfaces 2.1.8. Simulate, Compile and Validate the Design on Hardware
2.2. Reset Sequence x
2.2.1. Cold Reset Sequence Triggered by PERST# Signal 2.2.2. Warm Reset Sequence Triggered by Hot Reset 2.2.3. Cold Reset Triggered by User 2.2.4. Warm Reset Triggered by User
3. Configuring and Generating the GTS AXI Multichannel DMA IP for PCI Express x
3.1. Creating the Quartus® Prime Project 3.2. Configuring the GTS AXI Multichannel DMA IP for PCI Express 3.3. Generating HDL for Simulation and Synthesis 3.4. Generating the Design Example 3.5. Compiling the Design Example
3.2. Configuring the GTS AXI Multichannel DMA IP for PCI Express x
3.2.1. IP Settings 3.2.2. PCIe Settings
3.2.2. PCIe Settings x
3.2.2.1. MCDMA Settings 3.2.2.2. PCI Express / PCI Capabilities 3.2.2.3. Base Address Registers
3.2.2.1. MCDMA Settings x
3.2.2.1.1. Multichannel DMA Mode 3.2.2.1.2. Bursting Master Mode 3.2.2.1.3. Bursting Slave Mode 3.2.2.1.4. BAM + BAS Mode 3.2.2.1.5. BAM + MCDMA Mode 3.2.2.1.6. BAM + BAS + MCDMA Mode
3.2.2.2. PCI Express / PCI Capabilities x
3.2.2.2.1. PCIe Device 3.2.2.2.2. PCIe MSI-X
3.2.2.2.2. PCIe MSI-X x
3.2.2.2.2.1. PCIe PF MSI-X 3.2.2.2.2.2. PCIe VF MSI-X
3.3. Generating HDL for Simulation and Synthesis x
3.3.1. IP Core Generation Output
4. Integrating the IP With Your Application x
4.1. Overview 4.2. Implementing Required Clocking 4.3. Implementing Required Resets 4.4. Connecting the GTS AXI Streaming IP-Facing Interfaces 4.5. Connect User Logic-Facing Interfaces
4.4. Connecting the GTS AXI Streaming IP-Facing Interfaces x
4.4.1. PCIe AXI-Stream TX Interface (ss_tx_st) 4.4.2. PCIe AXI-Stream RX Interface (ss_rx_st) 4.4.3. Control and Status Register Interface (ss_csr_lite) 4.4.4. Transmit Flow Control Credit Interface (ss_txcrdt) 4.4.5. Configuration Intercept Interface (CII) 4.4.6. Completion Timeout Interface (ss_cplto) 4.4.7. Function Level Reset (FLR) Interface 4.4.8. Control Shadow Interface (ss_ctrlshadow) 4.4.9. Error Interface
4.4.5. Configuration Intercept Interface (CII) x
4.4.5.1. Configuration Intercept Request Interface (ss_cii_req) 4.4.5.2. Configuration Intercept Response Interface (ss_cii_resp)
4.4.7. Function Level Reset (FLR) Interface x
4.4.7.1. FLR Received Interface (ss_flrrcvd) 4.4.7.2. FLR Completion Interface (ss_flrcmpl)
4.5. Connect User Logic-Facing Interfaces x
4.5.1. H2D AXI-Stream Manager (h2d_st_initatr) 4.5.2. D2H AXI-Stream Subordinate (d2h_st_respndr) 4.5.3. H2D/D2H AXI-MM Manager (dma_mm_initatr) 4.5.4. BAM AXI-MM Manager (bam_mm_initatr) 4.5.5. BAS AXI-MM Subordinate (bas_mm_respndr) 4.5.6. PIO AXI-Lite Manager (pio_lite_initiatr) 4.5.7. HIP Reconfiguration AXI-Lite Subordinate (user_csr_lite) 4.5.8. User Event MSI-X (user_msix) 4.5.9. User Event MSI (user_msi) 4.5.10. User Function Level Reset (user_flr) 4.5.11. User Configuration Intercept Interface 4.5.12. Configuration Slave (cs_lite_respndr)
4.5.11. User Configuration Intercept Interface x
4.5.11.1. User Configuration Intercept Request Interface (user_cii_req) 4.5.11.2. User Configuration Intercept Response Interface (user_cii_resp)
5. Simulating the IP x
5.1. Design Example Overview 5.2. Design Example Components 5.3. Simulating the Design Example
5.1. Design Example Overview x
5.1.1. AXI-S Device-side Packet Loopback Design Example
6. Validating the IP x
6.1. Hardware and Software Requirements 6.2. Running the Design Example Application on a Hardware Setup
6.2. Running the Design Example Application on a Hardware Setup x
6.2.1. Program the FPGA 6.2.2. Configuration Changes from BIOS 6.2.3. Host Operating System Check (if Working with Ubuntu 22.04) 6.2.4. Installing the Required Kernel Version (if Working with Ubuntu 24.04) 6.2.5. Set the Boot Parameters 6.2.6. Custom Driver 6.2.7. DPDK Poll Mode Driver
6.2.6. Custom Driver x
6.2.6.1. Prerequisites 6.2.6.2. PIO Test 6.2.6.3. Custom PIO Read Write Test 6.2.6.4. DMA Test
6.2.6.1. Prerequisites x
6.2.6.1.1. Steps for UIO Driver Installation 6.2.6.1.2. Enable Virtual Function 6.2.6.1.3. Build and Install User Space Library 6.2.6.1.4. Build the Reference Application
6.2.6.4. DMA Test x
6.2.6.4.1. AXI-S Device-side Packet Loopback (Device-side Packet Loopback)
6.2.7. DPDK Poll Mode Driver x
6.2.7.1. Prerequisites 6.2.7.2. PIO Test 6.2.7.3. DMA Test
6.2.7.1. Prerequisites x
6.2.7.1.1. Install PMD Driver 6.2.7.1.2. Bind the Device 6.2.7.1.3. Enable Virtual Function
6.2.7.3. DMA Test x
6.2.7.3.1. Device-side Packet Loopback
A. Appendix A: Functional Description x
A.1. High-level Functional Overview
A.1. High-level Functional Overview x
A.1.1. Multichannel DMA A.1.2. Bursting Master (BAM) A.1.3. Bursting Slave (BAS) A.1.4. MSI Interrupt A.1.5. Configuration Slave (CS) A.1.6. Root Port Address Translation Table Enablement A.1.7. Control and Status Register Interface A.1.8. Configuration Intercept Interface
A.1.1. Multichannel DMA x
A.1.1.1. H2D Data Mover A.1.1.2. D2H Data Mover A.1.1.3. Descriptors A.1.1.4. AXI4-Lite PIO Manager A.1.1.5. AXI-MM Write (H2D) and Read (D2H) Manager A.1.1.6. AXI-Stream Manager (H2D) and Subordinate (D2H) A.1.1.7. User MSI-X A.1.1.8. User Function Level Reset (FLR) A.1.1.9. Control and Status Registers
A.1.1.2. D2H Data Mover x
A.1.1.2.1. D2H Descriptor Fetch
A.1.1.3. Descriptors x
A.1.1.3.1. Support for Unaligned (or Byte-Aligned) Data Transfer A.1.1.3.2. Metadata Support A.1.1.3.3. MSI-X/Writeback
A.1.4. MSI Interrupt x
A.1.4.1. Endpoint MSI Support Through BAS A.1.4.2. MSI Interrupt Controller
A.1.5. Configuration Slave (CS) x
A.1.5.1. 14-bit AXI-MM Address Format A.1.5.2. Configuration Access Mechanism
B. Appendix B: Registers x
B.1. Queue Control B.2. MSI-X Memory Space B.3. Control Register (GCSR)
1. Overview
2. Quick Start Guide
3. Configuring and Generating the GTS AXI Multichannel DMA IP for PCI Express
4. Integrating the IP With Your Application
5. Simulating the IP
6. Validating the IP
A. Appendix A: Functional Description
B. Appendix B: Registers
C. Document Revision History for the GTS AXI Multichannel DMA IP for PCI Express*

B.1. Queue Control

The QCSR space contains queue control and status information. This register space of 1 MB can support up to 256 H2D and 256 D2H queues, where each queue is allocated 256 bytes of register space. The memory space allocated to each function is enough for each function to have allocated all the DMA Channels. However, the actual number depends on the parameters input at IP generation time.

Address [7:0]: Registers for the queues

Address [18:8]: Queue number

Address [19]: 0 = D2H, 1=H2D

The following registers are defined for the H2D/D2H queues. The base addresses for H2D and D2H are different, but registers (H2D and D2H) have the same address offsets.

Table 71.  Queue Control Registers
Register Name Address Offset Access Type Description
Q_CTRL 8’h00 R/W Control Register
RESERVED 8’h04   Reserved
Q_START_ADDR_L 8’h08 R/W Lower 32 bits of the queue base address in system memory. This is the beginning of the linked-list of 4KB pages containing the descriptors.
Q_START_ADDR_H 8’h0C R/W Upper 32 bits of the queue base address in system memory. This is the beginning of the linked-list of 4KB pages containing the descriptors.
Q_SIZE 8’h10 R/W Number of maximum entries in a queue. Powers of 2 only.
Q_TAIL_POINTER 8’h14 R/W Current pointer to the last valid descriptor queue entry in the host memory.
Q_HEAD_POINTER 8’h18 RO Current pointer to the last descriptor that was fetched. Updated by the Descriptor Fetch Engine.
Q_COMPLETED_POINTER 8’h1C RO Last completed pointer after DMA is done. Software can poll this for status if Writeback is disabled.
Q_CONSUMED_HEAD_ADDR_L 8’h20 R/W Lower 32 bits of the system address where the ring consumed pointer is stored. This address is used for consumed pointer writeback.
Q_CONSUMED_HEAD_ADDR_H 8’h24 R/W Upper 32 bits of the system address where the ring consumed pointer is stored. This address is used for consumed pointer writeback.
Q_BATCH_DELAY 8’h28 R/W Delay the descriptor fetch until the time elapsed from a prior fetch exceeds the delay value in this register to maximize fetching efficiency.
Q_DATA_DRP_ERR_CTR 8’h40 RW Data drop error counter.
Q_PYLD_CNT 8’h44 R/W 20-bit payload count. DMA payload size in bytes and must be 64-byte aligned. Maximum 1 MB, with 20’h0 indicating 1 MB. The value set in this register must be the same as used by the Host software to populate the PYLD_CNT field of descriptors for the respective channel. Applicable only for D2H AXI-S port mode. Unused in all other modes.
Q_RESET 8’h48 R/W Request reset for the queue by writing 1’b1 to this register, and poll for a value of 1’b0 when reset has been completed by hardware. Hardware clears this bit after completing the reset of a queue. Similar process occurs when FLR reset is detected for a VF.

The following registers are defined for each implemented H2D and D2H queue. The total QCSR address space for each H2D/D2H is 256B and requires 8 bits of address.

Table 72.  Q_CTRL (Offset 8'h00)
Bits[31:0] Name R/W Default Description
[31:10] rsvd     Reserved
[9] q_intr_en R/W 0 If set, upon completion, generate an MSI-X interrupt for the queue.
[8] q_wb_en R/W 0 If set, upon completion, do a write-back for the queue.
[7:1] rsvd     Reserved
[0] q_en R/W 0 Enable. Once it is enabled, the DMA starts fetching pending descriptors and executing them.
Table 73.  Q_START_ADDR_L (Offset 8'h08)
Bits[31:0] Name R/W Default Description
[31:0] q_strt_addr_l R/W 0 After software allocates the descriptor ring buffer, it writes the lower 32-bit allocated address to this register. The descriptor fetch engine uses this address and the pending head/tail pointer to fetch the descriptors.
Table 74.  Q_START_ADDR_H (Offset 8'h0C)
Bits[31:0] Name R/W Default Description
[31:0] q_strt_addr_h R/W 0 After software allocates the descriptor ring buffer, it writes the upper 32-bit allocated address to this register. The descriptor fetch engine uses this address and the pending head/tail pointer to fetch the descriptors.
Table 75.  Q_SIZE (Offset 8'h10)
Bits[31:0] Name R/W Default Description
[31:5] rsvd     Reserved
[4:0] q_size R/W 1 Size of the descriptor ring in power of 2 and maximum value of 16. The unit is number of descriptors. Hardware defaults to using a value of 1 if an illegal value is written. A value of 1 means a queue size of 2 (2^1). A value of 16 (0x10) means a queue size of 64K (2^16).
Table 76.  Q_TAIL_POINTER (Offset 8'h14)
Bits[31:0] Name R/W Default Description
[31:16] rsvd     Reserved
[15:0] q_tl_ptr R/W 0

After software sets up a last valid descriptor in the descriptor buffer, it programs this register with the position of the last (tail) valid descriptor that is ready to be executed.

The DMA Descriptor Engine fetches descriptors from the buffer up to this position of the buffer.

Note: Writing 0x0 to Q_TAIL_POINTER is illegal.
Table 77.  Q_HEAD_POINTER (Offset 8'h18)
Bits[31:0] Name R/W Default Description
[31:25] rsvd     Reserved
[24] q_err_during_desc_fetch RO 0 Error during descriptor fetch, e.g. Cmplto/UR.
[23:16] rsvd Reserved
[15:0] q_hd_ptr R/W 0 After the DMA Descriptor Fetch Engine fetches the descriptors from the descriptor buffer, up to the tail pointer, it updates this register with that last fetched descriptor position. The fetch engine only fetches descriptors if the head and tail pointers are not equal.
Table 78.  Q_COMPLETED_POINTER (Offset 8'h1C)
Bits[31:0] Name R/W Default Description
[31:16] rsvd     Reserved
[15:0] q_cmpl_ptr R/W 0 This register is updated by hardware to store the last descriptor position (pointer) that DMA has completed, and all data for that descriptor and previous descriptors has arrived at the intended destinations.

Software can poll this register to find out the status of the DMA for a specific queue.
Table 79.  Q_CONSUMED_HEAD_ADDR_L (Offset 8'h20)
Bits[31:0] Name R/W Default Description
[31:0] q_cnsm_hd_addr_l R/W 0 Software programs this register with the lower 32-bit address location where the writeback targets after DMA is completed for a descriptor with writeback bit enabled.
Table 80.  Q_CONSUMED_HEAD_ADDR_H (Offset 8'h24)
Bits[31:0] Name R/W Default Description
[31:0] q_cnsm_hd_addr_h R/W 0 Software programs this register with the upper 32-bit address location where the writeback targets after DMA is completed for a set of descriptors.
Table 81.  Q_BATCH_DELAY (Offset 8'h28)
Bits[31:0] Name R/W Default Description
[31:20] rsvd     Reserved
[19:0] q_batch_dscr_delay R/W 0 Software programs this register with the amount of time between fetches for descriptors. Each unit is 2ns.
Table 82.  Q_DATA_DRP_ERR_CTR (Offset 8'h40)
Bits[31:0] Name R/W Default Description
[31:21] rsvd     Reserved
[20] q_d2h_data_drop_err_st R/W 0

D2H data drop error status for a channel.

This bit is set when packets are received for a channel that is not enabled or when packets are dropped for a channel that is enabled.

[19] rsvd     Reserved
[18] rsvd     Reserved
[17] q_data_err_mm R/W 0 Data error status for a channel in AXI-MM mode.
[16] q_data_err_st R/W 0 Data error status for a channel in AXI-S mode.
[15:0] q_data_drp_err_cnt R/W 0

Data drop error count. The error count is updated by the hardware and count increments whenever the D2H side drops a packet. The error count saturates at all 1's and needs software to clear.

Data drop example cases:

  • Packets received for a channel that is not enabled.
  • Packets received for a channel that is enabled, but does not have TID update.
Note: For packets received for a channel that is enabled, TID update is done by software. However, the descriptors that are not yet fetched are not dropped.
Table 83.  Q_PYLD_CNT (Offset 8'h44)
Bits[31:0] Name R/W Default Description
[31:20] rsvd     Reserved
[19:0] q_pyld_cnt R/W 0 20-bit payload count. DMA payload size in bytes. Maximum 1 MB, with 20’h0 indicating 1 MB. This value must be the same as set in the descriptors payload count field. Applicable only for D2H AXI-S port mode. Unused in all other modes.
Table 84.  Q_RESET (Offset 8'h48)
Bits[31:0] Name R/W Default Description
[31:1] rsvd     Reserved
[0] q_reset R/W 0 Request reset for the queue by writing 1’b1 to this register, and poll for a value of 1’b0 when reset has been completed by hardware. Hardware clears this bit after completing the reset of a queue.
Level Two Title