Scalable Scatter-Gather DMA Intel® FPGA IP Design Example User Guide

Download PDF
ID 823179
Date 4/30/2025
Public
Document Table of Contents
Document Table of Contents x
1. Design Example Description 2. Quick Start Guide 3. Document Revision History for the Scalable Scatter-Gather DMA Intel® FPGA IP Design Example User Guide
1. Design Example Description x
1.1. H2D ST to D2H ST Loopback Design Example
1.1. H2D ST to D2H ST Loopback Design Example x
1.1.1. DMA PCIe* Mode Design Example 1.1.2. DMA SoC Mode Design Example
2. Quick Start Guide x
2.1. Directory Structure 2.2. Generating the Design Example 2.3. Simulating the Design Example 2.4. Testbench 2.5. Compiling the Design Example 2.6. Hardware and Software Requirements 2.7. Running the Design Example
2.3. Simulating the Design Example x
2.3.1. DMA SoC Mode Example Design 2.3.2. DMA PCIe Mode Example Design
2.3.2. DMA PCIe Mode Example Design x
2.3.2.1. Running Simulation using VCS* MX 2.3.2.2. Running Simulation Using QuestaSim* 2.3.2.3. Running Simulation using Xcelium 2.3.2.4. Running Simulation using Riviera-PRO
2.4. Testbench x
2.4.1. DMA SoC Mode H2D ST to D2H ST Loopback Example Testbench 2.4.2. DMA PCIe Mode H2D ST to D2H ST Loopback & H2D MM Example Testbench
2.4.2. DMA PCIe Mode H2D ST to D2H ST Loopback & H2D MM Example Testbench x
2.4.2.1. Design Example Testbench
2.7. Running the Design Example x
2.7.1. Running the DMA SoC Mode H2D ST to D2H ST Loopback Design Example 2.7.2. Running the DMA PCIe Mode H2D ST to D2H ST Loopback Design Example
2.7.1. Running the DMA SoC Mode H2D ST to D2H ST Loopback Design Example x
2.7.1.1. Building the Software Application Project for Nios® V/m Processor 2.7.1.2. Programming the Example Design into the Development Board
2.7.2. Running the DMA PCIe Mode H2D ST to D2H ST Loopback Design Example x
2.7.2.1. Installing the Linux Kernel Driver 2.7.2.2. Running the Loopback Design Example
2.7.2.2. Running the Loopback Design Example x
2.7.2.2.1. Testing H2D ST to D2H ST Loopback 2.7.2.2.2. Testing H2D MM
1. Design Example Description
2. Quick Start Guide
3. Document Revision History for the Scalable Scatter-Gather DMA Intel® FPGA IP Design Example User Guide

2.4.2. DMA PCIe Mode H2D ST to D2H ST Loopback & H2D MM Example Testbench

The simulation testbench instantiates the SSGDMA DMA PCIe Mode loopback design example and a Root Port BFM to interface with the target Endpoint.

The testbench uses a test driver module, altpcietb_bfm_rp_gen4_x16.sv, to initiate the configuration and exercise the target memory and DMA channel in the Endpoint. This testbench simulates the scenario of a single Root Port connected to a single Endpoint. At startup, the test driver module displays information from the Root Port and Endpoint Configuration Space registers. The Root Port BFM first sets up memory locations within the Host Shared memory. Data from the Host memory is then sent to the device port by the SSGDMA IP via H2D DMA operations. Finally, the IP loops this data back to the Host memory using D2H DMA operations.

The following figure presents a high-level view of the SSGDMA DMA PCIe Mode loopback design example.

Figure 14. Block Diagram for the SSGDMA DMA PCIe Mode Loopback Design Example Simulation Testbench

The top-level of the testbench instantiates the following main modules:

  • altpcietb_bfm_rp_gen4x16.sv - Root Port PCIe* BFM.

    Directory path:

    <example_design>/PCIE_MODE/pcie_ed_sim_tb/ip/pcie_ed_sim_tb/ dut_pcie_tb_ip/intel_pcie_gts_tbed_<ver>/sim
  • pcie_ed_dut.ip - Endpoint design with the parameters pre-defined as required by SSGDMA IP parameters.

    Directory path:

    <example_design>/PCIE_MODE/ip/pcie_ed
  • pcie_ed_ssgdma.ip - SSGDMA IP with the parameters that you specify.

    Directory path: 

    <example_design>/PCIE_MODE/ip/pcie_ed

The testbench starts with link training and then accesses the GTS AXI Streaming IP's configuration space for enumeration. A task called ssgdma_loopback_test_1port (defined in the Root Port PCIe BFM altpcietb_bfm_rp_gen4_x16.sv) performs the DMA operation.

This testbench implements the transfer of one packet with a length of 32 bytes. The DMA testbench for the design example completes the following tasks:

  1. Set up 32 bytes of incrementing data pattern to test data movement from the host to the device and back to the host.
  2. Enable and configure MSI-X to launch a memory write to signal the end of each descriptor’s DMA transaction. The Write-Back function is disabled for the simulation.
  3. Set up the H2D (Host-to-Device) ST Queue CSR of the SSGDMA IP.
  4. Set up one H2D ST descriptor in the host memory, with the source address pointing to the incrementing data pattern locations in the host memory. The descriptors indicate the start of packet (SOF) and end of packet (EOF) markers, along with packet length.
  5. At the last step of the Queue programming, the SSGDMA Q_INSERT_POINTER register is configured, which triggers the SSGDMA IP to start the H2D ST DMA transaction. Descriptors is fetched from the Host memory, and data read from the Host memory is forwarded to the respective device agent.
  6. Next, set up the D2H (Device-to-Host) ST Queue CSR of the SSGDMA IP.
  7. Enable and configure MSI-X to launch a memory write to signal the end of each descriptor’s DMA transaction. The Write-Back function is disabled for the simulation.
  8. Set up one D2H ST descriptor in the host memory, with the destination address pointing to a new address space in the host memory, which is pre-filled with all zeroes.
  9. At the last step of the Queue programming, the SSGDMA Q_INSERT_POINTER register is configured, which triggers the SSGDMA IP to start the D2H ST DMA transaction. Descriptors is fetched from the Host memory; once data is available at the respective device port, the data is written to the Host memory according to the descriptors.
  10. MSI-X write commands are triggered for every descriptor completion, and the testbench checks for proper receipt.
  11. Data written back to the Host memory in D2H transfer is compared with standard incrementing patterns to determine test success or failure.
  12. The simulation reports the following message if no errors occur. 
    “Simulation stopped due to successful completion”
Figure 15. H2D ST to D2H ST Loopback with Incremental Data Simulation Waveforms
Figure 16. Completion TLP from Root Port to SSGDMA IP Waveforms
Figure 17. Memory Write TLP from SSGDMA IP to Root Port Waveforms

The following sample output illustrates a successful simulation test run. 

# INFO:           76600 ns RP User Avmm Driver: begin RP Configuration.
# INFO:          122119 ns RP User Avmm Driver: Finished RP Configuration!
# INFO:          122119 ns Completed initial configuration of Root Port.
# INFO:          124799 ns   EP PCI Express Link Status Register (1043):
# INFO:          124799 ns     Negotiated Link Width: x4 
# INFO:          124799 ns         Slot Clock Config: System Reference Clock Used 
# INFO:          124799 ns            New Link Speed: 8.0GT/s 
# INFO:          124799 ns   
# INFO:          125415 ns   EP PCI Express Link Control Register (0000):
# INFO:          125415 ns       Common Clock Config: Local Clock Used 
# INFO:          125415 ns 
# INFO:          126143 ns 
# INFO:          126143 ns   EP PCI Express Capabilities Register (0002): 
# INFO:          126143 ns        Capability Version: 2 
# INFO:          126143 ns                 Port Type: Native Endpoint
# INFO:          126143 ns  
# INFO:          126143 ns   EP PCI Express Device Capabilities Register (00008021):
# INFO:          126143 ns     Max Payload Supported: 256 Bytes 
# INFO:          126143 ns              Extended Tag: Supported 
# INFO:          126143 ns    Acceptable L0s Latency: Less Than 64 ns 
# INFO:          126143 ns    Acceptable L1  Latency: Less Than 1 us  
# INFO:          126143 ns          Attention Button: Not Present 
# INFO:          126143 ns       Attention Indicator: Not Present 
# INFO:          126143 ns           Power Indicator: Not Present 
# INFO:          126143 ns    
# INFO:          126143 ns   EP PCI Express Link Capabilities Register (01400C43): 
# INFO:          126143 ns        Maximum Link Width: x4
# INFO:          126143 ns      Supported Link Speed: 8.0GT/s or 5.0GT/s or 2.5GT/s  
# INFO:          126143 ns                 L0s Entry: Supported 
# INFO:          126143 ns                 L1  Entry: Supported 
# INFO:          126143 ns          L0s Exit Latency: Less Than 64 ns 
# INFO:          126143 ns          L1  Exit Latency: Less Than 1 us
# INFO:          126143 ns               Port Number: 01  
# INFO:          126143 ns   Surprise Dwn Err Report: Not Supported   
# INFO:          126143 ns    DLL Link Active Report: Not Supported   
# INFO:          126143 ns                                         
# INFO:          126143 ns   EP PCI Express Device Capabilities 2 Register (00711B9F):   
# INFO:          126143 ns   Completion Timeout Rnge: ABCD (50us to 64s)   
# INFO:          126807 ns                                    
# INFO:          126807 ns   EP PCI Express Device Control Register (1130):  
# INFO:          126807 ns   Error Reporting Enables: 0    
# INFO:          126807 ns          Relaxed Ordering: Enabled    
# INFO:          126807 ns               Max Payload: 256 Bytes  
# INFO:          126807 ns              Extended Tag: Enabled    
# INFO:          126807 ns          Max Read Request: 256 Bytes   
# INFO:          126807 ns                                      
# INFO:          126807 ns   EP PCI Express Device Status Register (0010): 
# INFO:          126807 ns                                      
# INFO:          127423 ns   EP PCI Express Virtual Channel Capability: 
# INFO:          127423 ns          Virtual Channel: 1  
# INFO:          127423 ns          Low Priority VC: 0  
# INFO:          127423 ns     
# INFO:          128191 ns     
# INFO:          128191 ns Configuring Bus 001, Device 000, Function 00  
# INFO:          128191 ns   EP Read Only Configuration Registers:  
# INFO:          128191 ns                 Vendor ID: 1172 
# INFO:          128191 ns                 Device ID: 0005 
# INFO:          128191 ns               Revision ID: 01   
# INFO:          128191 ns                Class Code: FF0000    
# INFO:          128191 ns       Subsystem Vendor ID: 0000      
# INFO:          128191 ns              Subsystem ID: 0000      
# INFO:          128191 ns             Interrupt Pin: No INTx# pin used  
# INFO:          128191 ns                                        
# INFO:          128807 ns   PCI MSI Capability Register:         
# INFO:          128807 ns    64-Bit Address Capable: Not Supported  
# INFO:          128807 ns        Messages Requested:  1       
# INFO:          128807 ns                                    
# INFO:          130031 ns   PCI MSI-X Capability Register:    
# INFO:          130031 ns                Table Size: 000A    
# INFO:          130647 ns                 Table BIR: 0       
# INFO:          130647 ns              Table Offset: 00020000    
# INFO:          131263 ns                   PBA BIR: 0           
# INFO:          131263 ns                PBA Offset: 00030000    
# INFO:          131263 ns                                        
# INFO:          132487 ns   EP PCI Express AER Capability Register:   
# INFO:          134335 ns        ECRC Check Capable: Supported   
# INFO:          134335 ns   ECRC Generation Capable: Supported  
# INFO:          134335 ns                    
# INFO:          136639 ns                   
# INFO:          136639 ns BAR Address Assignments:  
# INFO:          136639 ns BAR    Size       Assigned Address  Type   
# INFO:          136639 ns ---    ----       ----------------         
# INFO:          136639 ns BAR0     4 MBytes          00400000 Non-Prefetchable  
# INFO:          136639 ns BAR1   Disabled                    
# INFO:          136639 ns BAR2   Disabled                   
# INFO:          136639 ns BAR3   Disabled                   
# INFO:          136639 ns BAR4   Disabled                    
# INFO:          136639 ns BAR5   Disabled                      
# INFO:          136639 ns ExpROM Disabled                       
# INFO:          137487 ns                                     
# INFO:          137487 ns Completed configuration of Endpoint BARs.    
# INFO:          138103 ns Issue Mwr/MRd to sSGDMA Loopback ED..   
#  
# INFO:          138000 ns Starting sSGDMA with 1 Ports in loopback...!!! #  
# INFO:          141000 ns Initializing RP Memory for DMA-RD  Channel-- 0 
# INFO:          141000 ns Initializing RP Memory for DMA-WR  Channel-- 0 
# INFO:          141719 ns RP Memory Initialization Done!                  
#  
# INFO:          141000 ns Performing Global Reset and Write 0x1234_CAFE to Scratch Reg 
# INFO:          152615 ns Global reset Done! 
#                                                                                   
# INFO:          153000 ns Scratch Reg readback from sSGDMA == 0x1234cafe  
#  
# INFO:          157000 ns ------------>NUM_DESCRIPTORS 1  
# INFO:          161000 ns H2D: Got MSIX interrupt Status for Channel 0 
# INFO:          163000 ns D2H: Got MSIX interrupt Status for Channel 0 
# INFO:          163351 ns Passed: 0032 same bytes in BFM mem addr 0x00004000 and 0x00001000  
# INFO:          163000 ns Finished comparison for Channel 0  
#  
#  
# INFO:          163000 ns Finished comparison for All 1 Channel !!! 
#  
#  
# INFO:          163000 ns Starting H2D_MM_WR_RD_OCM_test...!!! 
#  
# INFO:          164275 ns Initializing RP Memory for DMA-RD...    
# INFO:          164275 ns Initializing RP Memory for DMA-WR ....   
# INFO:          164275 ns RP Memory Initialization Done!          
#  
# INFO:          169000 ns Performing 0x9999_CAFE Write to Scratch Reg 
# INFO:          170000 ns Scratch Reg readback from sSGDMA == 0x9999cafe  
#  
# INFO:          185000 ns ------------>NUM_DESCRIPTORS 2  
# INFO:          185000 ns H2D MM : Got MSIX interrupt Status for H2D MM 
# INFO:          185000 ns H2D: Got H2D_MM DATA[0] !!!!!!!!!!!!!!! 0x0 
# INFO:          185000 ns H2D: Got H2D_MM DATA[1] !!!!!!!!!!!!!!! 0x1 
# INFO:          185000 ns H2D: Got H2D_MM DATA[2] !!!!!!!!!!!!!!! 0x2 
# INFO:          185000 ns H2D: Got H2D_MM DATA[3] !!!!!!!!!!!!!!! 0x3 
# INFO:          185000 ns H2D: Got H2D_MM DATA[4] !!!!!!!!!!!!!!! 0x4 
# INFO:          185000 ns H2D: Got H2D_MM DATA[5] !!!!!!!!!!!!!!! 0x5 
# INFO:          185000 ns H2D: Got H2D_MM DATA[6] !!!!!!!!!!!!!!! 0x6 
# INFO:          185000 ns H2D: Got H2D_MM DATA[7] !!!!!!!!!!!!!!! 0x7 
#  
# INFO:          185000 ns Finished comparison for All Memory Address !!! 
#  
# INFO:          185691 ns sSGDMA ED Wr/rd Completed..!!!   
# SUCCESS: Simulation stopped due to successful completion! 
# Simulation passed

Section Content
Design Example Testbench

Level Two Title