Multi Channel DMA Intel® FPGA IP for PCI Express* User Guide

Download PDF
ID 683821
Date 1/29/2024
Public

A newer version of this document is available. Customers should click here to go to the newest version.

Document Table of Contents
Document Table of Contents x
1. Before You Begin 2. Introduction 3. Functional Description 4. Interface Overview 5. Parameters (H-Tile) 6. Parameters (P-Tile) (F-Tile) (R-Tile) 7. Designing with the IP Core 8. Software Programming Model 9. Registers 10. Troubleshooting/Debugging 11. Multi Channel DMA Intel FPGA IP for PCI Express User Guide Archives 12. Revision History for the Multi Channel DMA Intel FPGA IP for PCI Express User Guide
1. Before You Begin x
1.1. Terms and Acronyms 1.2. Known Issues
2. Introduction x
2.1. Multi Channel DMA IP for PCI Express Features 2.2. Device Family Support 2.3. Recommended Speed Grades 2.4. Resource Utilization 2.5. Release Information
2.1. Multi Channel DMA IP for PCI Express Features x
2.1.1. Endpoint Mode 2.1.2. Root Port Mode
3. Functional Description x
3.1. Multi Channel DMA 3.2. Bursting Avalon-MM Master (BAM) 3.3. Bursting Avalon-MM Slave (BAS) 3.4. MSI Interrupt 3.5. Config Slave (CS) 3.6. Root Port Address Translation Table Enablement 3.7. Hard IP Reconfiguration Interface 3.8. Config TL Interface 3.9. Configuration Intercept Interface (EP Only) 3.10. Data Mover Only
3.1. Multi Channel DMA x
3.1.1. H2D Data Mover 3.1.2. D2H Data Mover 3.1.3. Descriptors 3.1.4. Avalon-MM PIO Master 3.1.5. Avalon-MM Write (H2D) and Read (D2H) Master 3.1.6. Avalon-ST Source (H2D) and Sink (D2H) 3.1.7. User MSI-X 3.1.8. User Functional Level Reset (FLR) 3.1.9. Control Registers
3.1.2. D2H Data Mover x
3.1.2.1. D2H Descriptor Fetch
3.1.3. Descriptors x
3.1.3.1. Support for Unaligned (or byte-aligned) Data Transfer 3.1.3.2. Metadata Support 3.1.3.3. MSI-X/Writeback
3.1.6. Avalon-ST Source (H2D) and Sink (D2H) x
3.1.6.1. Avalon-ST 1-Port Mode 3.1.6.2. Packet (File) Boundary 3.1.6.3. Metadata
3.4. MSI Interrupt x
3.4.1. Endpoint MSI Support through BAS 3.4.2. MSI Interrupt Controller
3.5. Config Slave (CS) x
3.5.1. 29 bit AVMM address format 3.5.2. 14 bit AVMM address format 3.5.3. Configuration Access Mechanism
3.10. Data Mover Only x
3.10.1. H2D Data Mover 3.10.2. D2H Data Mover 3.10.3. Application Specific Bits
3.10.1. H2D Data Mover x
3.10.1.1. H2D Descriptor Format (h2ddm_desc) 3.10.1.2. H2D Descriptor Completion Packet Format (h2ddm_desc_cmpl) 3.10.1.3. H2D Descriptor Status (h2ddm_desc_status)
3.10.2. D2H Data Mover x
3.10.2.1. D2H Descriptor Format (d2hdm_desc) 3.10.2.2. D2H Descriptor Status (d2hdm_desc_status)
4. Interface Overview x
4.1. Port List 4.2. Clocks 4.3. Resets 4.4. Multi Channel DMA 4.5. Bursting Avalon-MM Master (BAM) Interface 4.6. Bursting Avalon-MM Slave (BAS) Interface 4.7. MSI Interface 4.8. Config Slave Interface (RP only) 4.9. Hard IP Reconfiguration Interface 4.10. Config TL Interface 4.11. Configuration Intercept Interface (EP Only) 4.12. Data Mover Interface 4.13. Hard IP Status Interface 4.14. Precision Time Management (PTM) Interface
4.1. Port List x
4.1.1. Port List (H-Tile) 4.1.2. Port List (P-Tile) (F-Tile) (R-Tile)
4.2. Clocks x
4.2.1. F-Tile System PLL Reference Clock Requirements
4.4. Multi Channel DMA x
4.4.1. Avalon-MM PIO Master 4.4.2. Avalon-MM Write Master (H2D) 4.4.3. Avalon-MM Read Master (D2H) 4.4.4. Avalon-ST Source (H2D) 4.4.5. Avalon-ST Sink (D2H) 4.4.6. User Event MSI-X Interface 4.4.7. User Functional Level Reset (FLR) Interface
4.12. Data Mover Interface x
4.12.1. H2D Data Mover Interface 4.12.2. D2H Data Mover Interface
5. Parameters (H-Tile) x
5.1. IP Settings 5.2. Example Designs
5.1. IP Settings x
5.1.1. System Settings 5.1.2. MCDMA Settings 5.1.3. Device Identification Registers 5.1.4. Multifunction and SR-IOV System Settings Parameters [Endpoint Mode] 5.1.5. Configuration, Debug and Extension Options 5.1.6. PHY Characteristics 5.1.7. PCI Express / PCI Capabilities Parameters
5.1.7. PCI Express / PCI Capabilities Parameters x
5.1.7.1. Device 5.1.7.2. Link 5.1.7.3. MSI-X 5.1.7.4. Power Management 5.1.7.5. Vendor Specific Extended Capability (VSEC)
6. Parameters (P-Tile) (F-Tile) (R-Tile) x
6.1. Top-Level Settings 6.2. PCIe0 Settings 6.3. Example Designs 6.4. Analog Parameters (F-Tile MCDMA IP Only) 6.5. PCIe1 Settings
6.2. PCIe0 Settings x
6.2.1. Base Address Register 6.2.2. PCIe0 Configuration, Debug and Extension Options 6.2.3. PCIe0 Device Identification Registers 6.2.4. PCIe0 PCI Express / PCI Capabilities 6.2.5. MCDMA Settings
6.2.3. PCIe0 Device Identification Registers x
6.2.3.1. PCIe0 PF0 IDs 6.2.3.2. PCIe0 PF0 VF IDs
6.2.4. PCIe0 PCI Express / PCI Capabilities x
6.2.4.1. PCIe0 Device 6.2.4.2. PCIe0 Link 6.2.4.3. PCIe0 MSI-X 6.2.4.4. PCIe0 DEV SER 6.2.4.5. PCIe0 PRS 6.2.4.6. PCIe0 PTM 6.2.4.7. PCIe0 ACS Capabilities 6.2.4.8. PCIe0 ATS 6.2.4.9. PCIe0 TPH 6.2.4.10. Receiver Detection (F-Tile MCDMA IP Only)
6.2.4.7. PCIe0 ACS Capabilities x
6.2.4.7.1. PCIe0 ACS for Physical Functions 6.2.4.7.2. PCIe0 ACS for Virtual Functions
6.2.4.8. PCIe0 ATS x
6.2.4.8.1. PCIe0 ATS for Physical Functions 6.2.4.8.2. PCIe0 ATS for Virtual Functions
6.2.4.9. PCIe0 TPH x
6.2.4.9.1. PCIe0 TPH for Physical Functions 6.2.4.9.2. PCIe0 TPH for Virtual Functions
6.5. PCIe1 Settings x
6.5.1. PCIe1 Configuration, Debug and Extension Options
7. Designing with the IP Core x
7.1. Generating the IP Core 7.2. Simulating the IP Core 7.3. IP Core Generation Output - Intel® Quartus® Prime Pro Edition 7.4. Systems Integration and Implementation
7.4. Systems Integration and Implementation x
7.4.1. Required Supporting IP
8. Software Programming Model x
8.1. Multi Channel DMA IP Custom Driver 8.2. Multi Channel DMA IP DPDK Poll-Mode based Driver 8.3. Multi Channel DMA IP Kernel Mode Network Device Driver
8.1. Multi Channel DMA IP Custom Driver x
8.1.1. Architecture 8.1.2. libmqdma library details 8.1.3. Application 8.1.4. Software Flow 8.1.5. API Flow 8.1.6. libmqdma Library API List 8.1.7. Request Structures
8.1.2. libmqdma library details x
8.1.2.1. Channel Initialization 8.1.2.2. Descriptor Memory Management 8.1.2.3. Descriptor Completion Status Update
8.1.5. API Flow x
8.1.5.1. Single Descriptor Load and Submit 8.1.5.2. Multiple Descriptor Load and Submit
8.1.6. libmqdma Library API List x
8.1.6.1. ifc_api_start 8.1.6.2. ifc_mcdma_port_by_name 8.1.6.3. ifc_qdma_device_get 8.1.6.4. ifc_num_channels_get 8.1.6.5. ifc_qdma_channel_get 8.1.6.6. ifc_qdma_acquire_channels 8.1.6.7. ifc_qdma_release_all_channels 8.1.6.8. ifc_qdma_device_put 8.1.6.9. ifc_qdma_channel_put 8.1.6.10. ifc_qdma_completion_poll 8.1.6.11. ifc_qdma_request_start 8.1.6.12. ifc_qdma_request_prepare 8.1.6.13. ifc_qdma_descq_queue_batch_load 8.1.6.14. ifc_qdma_request_submit 8.1.6.15. ifc_qdma_pio_read32 8.1.6.16. ifc_qdma_pio_write32 8.1.6.17. ifc_qdma_pio_read64 8.1.6.18. ifc_qdma_pio_write64 8.1.6.19. ifc_qdma_pio_read128 8.1.6.20. ifc_qdma_pio_write128 8.1.6.21. ifc_qdma_pio_read256 8.1.6.22. ifc_qdma_pio_write256 8.1.6.23. ifc_request_malloc 8.1.6.24. ifc_request_free 8.1.6.25. ifc_app_stop 8.1.6.26. ifc_qdma_poll_init 8.1.6.27. ifc_qdma_poll_add 8.1.6.28. ifc_qdma_poll_wait 8.1.6.29. ifc_mcdma_port_by_name
8.2. Multi Channel DMA IP DPDK Poll-Mode based Driver x
8.2.1. Architecture 8.2.2. MCDMA Poll Mode Driver 8.2.3. DPDK based application 8.2.4. Request Structures 8.2.5. Software Flow 8.2.6. API Flow 8.2.7. API List
8.2.2. MCDMA Poll Mode Driver x
8.2.2.1. Completion Status Update 8.2.2.2. Metadata Support 8.2.2.3. User MSI-X
8.3. Multi Channel DMA IP Kernel Mode Network Device Driver x
8.3.1. Architecture 8.3.2. Driver Information 8.3.3. Software Flow
8.3.2. Driver Information x
8.3.2.1. Device Management 8.3.2.2. Channel Management 8.3.2.3. Descriptor Memory Management 8.3.2.4. Completions Management 8.3.2.5. ethtool support 8.3.2.6. debugfs support 8.3.2.7. SRIOV Support
8.3.3. Software Flow x
8.3.3.1. Host to Device Flow 8.3.3.2. Device to Host Flow
9. Registers x
9.1. Queue Control (QCSR) 9.2. MSI-X Memory Space 9.3. Control Register (GCSR)
10. Troubleshooting/Debugging x
10.1. Debug Toolkit
10.1. Debug Toolkit x
10.1.1. Overview 10.1.2. Enabling the Debug Toolkit 10.1.3. Launching the Debug Toolkit 10.1.4. Using the Debug Toolkit
10.1.4. Using the Debug Toolkit x
10.1.4.1. Main View 10.1.4.2. Toolkit Parameters 10.1.4.3. Channel Parameters 10.1.4.4. Eye Viewer 10.1.4.5. Link Inspector
10.1.4.4. Eye Viewer x
10.1.4.4.1. Running Eye Viewer in the P-Tile Debug Toolkit 10.1.4.4.2. Running Eye Viewer in the F-Tile Debug Toolkit
1. Before You Begin
2. Introduction
3. Functional Description
4. Interface Overview
5. Parameters (H-Tile)
6. Parameters (P-Tile) (F-Tile) (R-Tile)
7. Designing with the IP Core
8. Software Programming Model
9. Registers
10. Troubleshooting/Debugging
11. Multi Channel DMA Intel FPGA IP for PCI Express User Guide Archives
12. Revision History for the Multi Channel DMA Intel FPGA IP for PCI Express User Guide

9.1. Queue Control (QCSR)

QCSR space contains queue control and status information. This register space of 1 MB can support up to 2048 H2D and 2048 D2H queues, where each queue is allocated 256 bytes of register space. The memory space allocated to each function is enough for each function to have allocated all the DMA Channels. However, the actual number depends on the parameters input at IP generation time.

Address [7:0] : Registers for the queues

Address [18:8]: Queue number

Address [19]: 0 = D2H, 1=H2D

The following registers are defined for H2D/D2H queues. The base address for H2D and D2H are different, but registers (H2D and D2H) has the same address offsets.

Table 145.  Queue Control Registers
Register Name Address Offset Access Type Description
Q_CTRL 8’h00 R/W Control Register
RESERVED 8’h04 RESERVED
Q_START_ADDR_L 8’h08 R/W Lower 32-bit of queue base address in system memory. This is the beginning of the linked-list of 4KB pages containing the descriptors.
Q_START_ADDR_H 8’h0C R/W Upper 32-bit of queue base address in system memory. This is the beginning of the linked-list of 4KB pages containing the descriptors.
Q_SIZE 8’h10 R/W Number of max entries in a queue. Powers of 2 only.
Q_TAIL_POINTER 8’h14 R/W Current pointer to the last valid descriptor queue entry in the host memory.
Q_HEAD_POINTER 8’h18 RO Current pointer to the last descriptor that was fetched. Updated by Descriptor Fetch Engine.
Q_COMPLETED_POINTER 8’h1C RO Last completed pointer after DMA is done. Software can poll this for status if Writeback is disabled.
Q_CONSUMED_HEAD_ADDR_L 8’h20 R/W Lower 32-bit of the system address where the ring consumed pointer is stored. This address is used for comsumed pointer writeback.
Q_CONSUMED_HEAD_ADDR_H 8’h24 R/W Upper 32-bit of the system address where the ring consumed pointer is stored. This address is used for comsumed pointer writeback.
Q_BATCH_DELAY 8’h28 R/W Delay the descriptor fetch until the time elapsed from a prior fetch exceeds the delayvalue in this register to maximize fetching efficiency.
RESERVED 8’h2C RESERVED
RESERVED 18’h30 RESERVED
RESERVED 8’h34 RESERVED
Q_DEBUG_STATUS_1 8’h38 RO RESERVED
Q_DEBUG_STATUS_2 8’h3C RO RESERVED
Q_DATA_DRP_ERR_CTR 8’h40 RW Data drop error counter
Q_PYLD_CNT 8’h44 R/W

20-bit payload count. DMA payload size in bytes and must be 64 byte aligned. Max 1 MB, with 20’h0 indicating 1 MB. The value set in this register must be the same as used by Host SW to populate the PYLD_CNT field of descriptors for the respective channel. Applicable only for D2H AVST 1 port mode. Unused in all other modes

Q_RESET 8’h48 R/W

Request reset for the queue by writing 1’b1 to this register, and poll for value of 1’b0 when reset has been completed by hardware. Hardware clears this bit after completing the reset of a queue. Similar process occurs when FLR reset is detected for a VF.

The following registers are defined for each implemented H2D and D2H queue. The total QCSR address space for each H2D/D2H is 256B and requires 8-bit of address.

Table 146.   Q_CTRL (Offset 8’h00)
Bit [31:0] Name R/W Default Description
[31:10] rsvd Reserved
[9] q_intr_en R/W 0 If set, upon completion generate a MSI-X interrupt.
[8] q_wb_en R/W 0 If set, upon completion, do a write back.
[7:1] rsvd Reserved
[0] q_en R/W 0 Enable. Once it is enabled, the DMA starts fetching pending descriptors and executing them.
Table 147.   Q_START_ADDR_L (Offset 8’h08)
Bit [31:0] Name R/W Default Description
[31:0] q_strt_addr_l R/W 0 After software allocate the descriptor ring buffer, it writes the lower 32-bit allocated address to this register. The descriptor fetch engine use this address and the pending head/tail pointer to fetch the descriptors.
Table 148.   Q_START_ADDR_H (Offset 8’h0C)
Bit [31:0] Name R/W Default Description
[31:0] q_strt_addr_h R/W 0 After software allocate the descriptor ring buffer, it writes the upper 32-bit allocated address to this register. The descriptor fetch engine use this address and the pending head/tail pointer to fetch the descriptors.
Table 149.   Q_SIZE (Offset 8’h10)
Bit [31:0] Name R/W Default Description
[31:5] rsvd Reserved
[4:0] q_size R/W 1 Size of the descriptor ring in power of 2 and max value of 16. The unit is number of descriptors. Hardware defaults to using a value of 1 if an illegal value is written. A value of 1 means queue size of 2 (2^1). A value is 16 (0x10) means queue size of 64K (2^16).
Table 150.   Q_TAIL_POINTER (Offset 8’h14)
Bit [31:0] Name R/W Default Description
[31:16] rsvd Reserved
[15:0] q_tl_ptr R/W 0

After software sets up a last valid descriptor in the descriptor buffer, it programs this register with the position of the last (tail) valid descriptor that is ready to be executed.

The DMA Descriptor Engine fetches descriptors from the buffer upto this position of the buffer.

Note: Writing 0x0 to Q_TAIL_POINTER is illegal.
Table 151.   Q_HEAD_POINTER (Offset 8’h18)
Bit [31:0] Name R/W Default Description
[31:16] rsvd

Reserved

Note: Bit 24 is not reserved. For bit [24] q_err_during_desc_fetch H2D : Cmplto/UR etc… D2H : Cmplto/UR etc…
[15:0] q_hd_ptr R/W 0 After DMA Descriptor Fetch Engine fetches the descriptors from the descriptor buffer, upto the tail pointer, it updates this register with that last fetched descriptor position. The fetch engine only fetches descriptors if the head and tail pointer is not equal.
Table 152.   Q_COMPLETED_POINTER (Offset 8’h1C)
Bit [31:0] Name R/W Default Description
[31:16] rsvd Reserved
[15:0] q_cmpl_ptr R/W 0 This register is updated by hardware to store the last descriptor position (pointer) that DMA has completed, that is all data for that descriptor and previous descriptors have arrived at the intended destinations. Software can poll this register to find out the status of the DMA for a specific queue.
Table 153.   Q_CONSUMED_HEAD_ADDR_L (Offset 8’h20)
Bit [31:0] Name R/W Default Description
[31:0] q_cnsm_hd_addr_l R/W 0 Software programs this register with the lower 32-bit address location where the writeback targets after DMA is completed for a descriptor with writeback bit enabled.
Table 154.   Q_CONSUMED_HEAD_ADDR_H (Offset 8’h24)
Bit [31:0] Name R/W Default Description
[31:0] q_cnsm_hd_addr_h R/W 0 Software programs this register with the upper 32-bit address location where the writeback targets after DMA is completed for a set of descriptors.
Table 155.   Q_BATCH_DELAY (Offset 8’h28)
Bit [31:0] Name R/W Default Description
[31:20] rsvd Reserved
[19:0] q_batch_dscr_delay R/W 0 Software programs this register with the amount of time between fetches for descriptors. Each unit is 2ns.
Table 156.  Q_DATA_DRP_ERR_CTR (Offset 8’h40)Note: The register Q_DATA_DRP_ERR_CTR is defined for D2H only and reserved for H2D.
Bit [31:0] Name R/W Default Description
[31:21] rsvd - - -
[20] q_d2h_data_drop_err_st R/W 0

D2H data drop error status for a channel.

This bit is set when packets are received for a channel that is not enabled or when packets are dropped for a channel that is enabled.

[19] rsvd - - -
[18] rsvd - - -
[17] q_data_err_mm R/W 0

Data error status for a channel in AVMM mode

[16] q_data_err_st R/W 0

Data error status for a channel in AVST mode

[15:0] q_data_drp_err_cnt R/W 0

Data drop error count. The error count is updated by the hardware and count increments whenever the D2H side drops a packet. The error count saturates at all 1s and needs software to clear.

Data drop example cases:
  • Packets received for a channel that is not enabled
  • Packets received for a channel that is enabled, but does not have TID update
Note: For packets received for a channel that is enabled, TID update is done by software. However, the descriptors that are not yet fetched, are not dropped.
Table 157.   Q_PYLD_CNT register (offset 8'h44)
Bit [31:0] Name R/W Default Description
[31:20] rsvd Reserved
[19:0] q_pyld_cnt R/W 0

20-bit payload count. DMA payload size in bytes. Max 1 MB, with 20’h0 indicating 1 MB. This value has to be same as set in the descriptors payload count field. Applicable only for D2H AVST 1 port mode. Unused in all other modes

Table 158.   Q_RESET (Offset 8’h48)
Bit [31:0] Name R/W Default Description
[31:1] rsvd Reserved
[0] q_reset R/W 0 Request reset for the queue by writing 1’b1 to this register, and poll for value of 1’b0 when reset has been completed by hardware. Hardware clears this bit after completing the reset of a queue.
Level Two Title