L-Tile and H-Tile Avalon® Memory-mapped+ Intel® FPGA IP for PCI Express* User Guide

Download PDF
ID 683527
Date 3/05/2024
Public
Document Table of Contents
Document Table of Contents x
1. Introduction 2. Quick Start Guide 3. Block and Interface Descriptions 4. Parameters 5. Designing with the IP Core 6. Registers 7. Design Example and Testbench 8. Document Revision History for Intel® L- and H-tile Avalon® Memory-mapped+ IP for PCI Express* User Guide A. Avalon-MM IP Variants Comparison B. Root Port BFM C. BFM Procedures and Functions D. Troubleshooting and Observing the Link Status E. Root Port Enumeration
1. Introduction x
1.1. Features 1.2. Device Family Support 1.3. Release Information 1.4. Recommended Speed Grades 1.5. Resource Utilization 1.6. Transceiver Tiles 1.7. Channel Availability
2. Quick Start Guide x
2.1. Design Components 2.2. Directory Structure 2.3. Generating the Design Example 2.4. Simulating the Design Example 2.5. Compiling the Design Example and Programming the Device 2.6. Installing the Linux Kernel Driver 2.7. Running the Design Example Application 2.8. Ensuring the Design Example Meets Timing Requirements
3. Block and Interface Descriptions x
3.1. Block Descriptions 3.2. Interface Descriptions
3.1. Block Descriptions x
3.1.1. Tags
3.2. Interface Descriptions x
3.2.1. Avalon-MM Interface Summary 3.2.2. Clocks and Resets 3.2.3. System Interfaces
3.2.1. Avalon-MM Interface Summary x
3.2.1.1. Read Data Mover (RDDM) Interfaces 3.2.1.2. Write Data Mover (WRDM) Interfaces 3.2.1.3. Bursting Avalon-MM Slave (BAS) Interface 3.2.1.4. Bursting Avalon-MM Master (BAM) Interface
3.2.1.1. Read Data Mover (RDDM) Interfaces x
3.2.1.1.1. Read Data Mover Avalon-MM Write Master and Conduit 3.2.1.1.2. Read Data Mover Avalon-ST Descriptor Sinks 3.2.1.1.3. Read Data Mover Status Avalon-ST Source
3.2.1.2. Write Data Mover (WRDM) Interfaces x
3.2.1.2.1. Write Data Mover Avalon-MM Read Master and Conduit 3.2.1.2.2. Write Data Mover Avalon-ST Descriptor Sinks 3.2.1.2.3. Write Data Mover Status Avalon-ST Source
3.2.1.4. Bursting Avalon-MM Master (BAM) Interface x
3.2.1.4.1. PCIe Address to Avalon-MM Address Mapping
3.2.3. System Interfaces x
3.2.3.1. Serial Data Interface 3.2.3.2. PIPE Interface 3.2.3.3. Interrupts 3.2.3.4. Configuration Output Interface 3.2.3.5. Flush Requests 3.2.3.6. Reconfiguration 3.2.3.7. Hard IP Status and Link Training Conduit 3.2.3.8. Bus Master Enable (BME) Conduit 3.2.3.9. Test Conduit
3.2.3.3. Interrupts x
3.2.3.3.1. Interrupt Signals Available when the PCIe Hard IP is an Endpoint 3.2.3.3.2. MSI
3.2.3.6. Reconfiguration x
3.2.3.6.1. PCIe Hard IP Reconfiguration 3.2.3.6.2. Transceiver Reconfiguration 3.2.3.6.3. PLL Reconfiguration
4. Parameters x
4.1. Avalon-MM Settings 4.2. Base Address Registers 4.3. Device Identification Registers 4.4. PCI Express and PCI Capabilities Parameters 4.5. Configuration, Debug and Extension Options 4.6. PHY Characteristics 4.7. Example Designs
4.4. PCI Express and PCI Capabilities Parameters x
4.4.1. Device Capabilities 4.4.2. Link Capabilities 4.4.3. MSI and MSI-X Capabilities 4.4.4. Power Management 4.4.5. Vendor Specific Extended Capability (VSEC)
5. Designing with the IP Core x
5.1. Generation 5.2. Simulation 5.3. IP Core Generation Output ( Quartus® Prime Pro Edition) 5.4. Channel Layout and PLL Usage
6. Registers x
6.1. Configuration Space Registers
6.1. Configuration Space Registers x
6.1.1. Register Access Definitions 6.1.2. PCI Configuration Header Registers 6.1.3. PCI Express Capability Structures 6.1.4. Intel Defined VSEC Capability Header 6.1.5. Uncorrectable Internal Error Status Register 6.1.6. Uncorrectable Internal Error Mask Register 6.1.7. Correctable Internal Error Status Register 6.1.8. Correctable Internal Error Mask Register
6.1.4. Intel Defined VSEC Capability Header x
6.1.4.1. Intel Defined Vendor Specific Header 6.1.4.2. Intel Marker 6.1.4.3. JTAG Silicon ID 6.1.4.4. User Configurable Device and Board ID
7. Design Example and Testbench x
7.1. Overview of the Example Design 7.2. Overview of the Testbench
7.1. Overview of the Example Design x
7.1.1. Block Descriptions 7.1.2. Programming Model for the Example Design 7.1.3. DMA Operations Using the Example Design
7.1.1. Block Descriptions x
7.1.1.1. DMA Controller (DMA Example Design Only) 7.1.1.2. Traffic Generator and Checker (BAS Example Design Only) 7.1.1.3. Avalon-MM Address to PCIe Address Mapping 7.1.1.4. BAR Interpreter
7.1.1.1. DMA Controller (DMA Example Design Only) x
7.1.1.1.1. Register Set 7.1.1.1.2. Deadlock Risk and Avoidance 7.1.1.1.3. Interrupts 7.1.1.1.4. Using the DMA Controller
7.1.1.2. Traffic Generator and Checker (BAS Example Design Only) x
7.1.1.2.1. Register Set
7.1.3. DMA Operations Using the Example Design x
7.1.3.1. Read DMA Example 7.1.3.2. Write DMA Example 7.1.3.3. Using the Traffic Generator (BAS Example Design Only) 7.1.3.4. Using the Traffic Checker (BAS Example Design Only)
B. Root Port BFM x
B.1. Root Port BFM Overview B.2. Issuing Read and Write Transactions to the Application Layer B.3. Configuration of Root Port and Endpoint B.4. Configuration Space Bus and Device Numbering B.5. BFM Memory Map
C. BFM Procedures and Functions x
C.1. ebfm_barwr Procedure C.2. ebfm_barwr_imm Procedure C.3. ebfm_barrd_wait Procedure C.4. ebfm_barrd_nowt Procedure C.5. ebfm_cfgwr_imm_wait Procedure C.6. ebfm_cfgwr_imm_nowt Procedure C.7. ebfm_cfgrd_wait Procedure C.8. ebfm_cfgrd_nowt Procedure C.9. BFM Configuration Procedures C.10. BFM Shared Memory Access Procedures C.11. BFM Log and Message Procedures C.12. Verilog HDL Formatting Functions
C.9. BFM Configuration Procedures x
C.9.1. ebfm_cfg_rp_ep Procedure C.9.2. ebfm_cfg_decode_bar Procedure
C.10. BFM Shared Memory Access Procedures x
C.10.1. Shared Memory Constants C.10.2. shmem_write Procedure C.10.3. shmem_read Function C.10.4. shmem_display Verilog HDL Function C.10.5. shmem_fill Procedure C.10.6. shmem_chk_ok Function
C.11. BFM Log and Message Procedures x
C.11.1. ebfm_display Verilog HDL Function C.11.2. ebfm_log_stop_sim Verilog HDL Function C.11.3. ebfm_log_set_suppressed_msg_mask Task C.11.4. ebfm_log_set_stop_on_msg_mask Verilog HDL Task C.11.5. ebfm_log_open Verilog HDL Function C.11.6. ebfm_log_close Verilog HDL Function
C.12. Verilog HDL Formatting Functions x
C.12.1. himage1 C.12.2. himage2 C.12.3. himage4 C.12.4. himage8 C.12.5. himage16 C.12.6. dimage1 C.12.7. dimage2 C.12.8. dimage3 C.12.9. dimage4 C.12.10. dimage5 C.12.11. dimage6 C.12.12. dimage7
D. Troubleshooting and Observing the Link Status x
D.1. Troubleshooting D.2. PCIe Link Inspector Overview
D.1. Troubleshooting x
D.1.1. Simulation Fails to Progress Beyond Polling.Active State D.1.2. Hardware Bring-Up Issues D.1.3. Link Training D.1.4. Use Third-Party PCIe Analyzer
D.2. PCIe Link Inspector Overview x
D.2.1. PCIe* Link Inspector Hardware
D.2.1. PCIe* Link Inspector Hardware x
D.2.1.1. Enabling the PCIe* Link Inspector D.2.1.2. Launching the PCIe* Link Inspector D.2.1.3. The PCIe* Link Inspector LTSSM Monitor D.2.1.4. Accessing the Configuration Space and Transceiver Registers D.2.1.5. Additional Status Commands
D.2.1.3. The PCIe* Link Inspector LTSSM Monitor x
D.2.1.3.1. LTSSM Monitor Registers D.2.1.3.2. ltssm_save2file <file_name> D.2.1.3.3. ltssm_file2console D.2.1.3.4. ltssm_debug_check D.2.1.3.5. ltssm_display <num_states> D.2.1.3.6. ltssm_save_oldstates
D.2.1.4. Accessing the Configuration Space and Transceiver Registers x
D.2.1.4.1. PCIe* Link Inspector Commands D.2.1.4.2. NPDME PLL and Channel Commands D.2.1.4.3. Register Address Map
D.2.1.5. Additional Status Commands x
D.2.1.5.1. Displaying PLL Lock and Calibration Status Registers
1. Introduction
2. Quick Start Guide
3. Block and Interface Descriptions
4. Parameters
5. Designing with the IP Core
6. Registers
7. Design Example and Testbench
8. Document Revision History for Intel® L- and H-tile Avalon® Memory-mapped+ IP for PCI Express* User Guide
A. Avalon-MM IP Variants Comparison
B. Root Port BFM
C. BFM Procedures and Functions
D. Troubleshooting and Observing the Link Status
E. Root Port Enumeration

3.1. Block Descriptions

Figure 10. Top Level Block Diagram
Note: The Config Slave module is only present in Root Port variations.

The PCIe Hard IP implements the Transaction, Data Link, and Physical layers stack required by the PCI Express* protocol. This stack allows the user application logic in the Stratix® 10 FPGA to interface with another device via a PCI Express* link.

While the PCIe* Hard IP uses a 500 MHz clock, other modules in the Bridge use a 250 MHz clock. The Width and Rate Adapter handles the necessary conversion between these two clock domains. An additional adapter converts between the embedded header format of the PCIe Hard IP and the separate header and data format used by the other modules in the Avalon® -MM Bridge.

The Avalon® -MM Bridge mainly consists of four leaf modules that interact with an Avalon® -ST Scheduler. The four leaf modules are:

  • Bursting Master: This module converts memory read and write TLPs received over the PCIe link into Avalon® -MM burst read and write transactions, and sends back CplD TLPs for read requests it receives.
  • Bursting Slave: This module converts Avalon-MM read and write transactions into PCIe memory read and write TLPs to be transmitted over the PCIe link. This module also processes the CplD TLPs received for the read requests it sent.
  • Write Data Mover: This module uses PCIe memory write TLPs and Avalon® -MM read transactions to move large amounts of data from your application logic in the Avalon® -MM space to the system memory in the PCIe space.
  • Read Data Mover: This module uses PCIe memory read TLPs and Avalon® -MM write transactions to move large amounts of data from the system memory in the PCIe space to the FPGA memory in the Avalon® -MM space.

The Avalon® -ST Scheduler serves two main functions. In the receive direction, this module distributes the received TLPs to the appropriate leaf module based on the TLP’s type, address and tag. In the transmit direction, this module takes TLPs ready for transmission from the leaf modules and schedules them for transfer to the Width and Rate Adapter to be forwarded to the PCIe Hard IP, based on the available credits. These data transfers follow the PCI Express* Specifications' ordering rules to avoid deadlocks.

Descriptors provided to the Data Movers through one of their Avalon® -ST sink interfaces control the data transfers. The Data Movers report the transfers’ status through their Avalon® -ST source interfaces.

The Config Slave converts single-cycle, 32-bit Avalon® -MM read and write transactions into PCIe configuration read and write TLPs (CfgRd0, CfgRd1, CfgWr0 and CfgWr1) to be sent over the PCIe link. This module also processes the completion TLPs (Cpl and CplD) it receives in return.

Note: The Config Slave module is only present in Root Port variations. Root Port mode is a future enhancement, and is not supported in this release of the Quartus® Prime Pro Edition.

The Response Reordering module assembles and reorders completion TLPs received over the PCIe link for the Bursting Slave and the Read Data Mover. It routes the completions based on their tag (see Tags for tag allocation information).

No re-ordering is necessary for the completions sent to the Config module as it only issues one request TLP at a time.

The FLR module detects Function-Level Resets received over the PCIe link and informs the application logic and the Scheduler module.
Note: The Function-Level Reset (FLR) capability is not currently available. It may be available in a future release of Quartus® Prime.

You can individually enable or disable the Bursting Master and Slave, and the Read and Write Data Mover modules by making GUI selections in any combination in the Parameters Editor inside the Quartus® Prime Pro Edition.

Endpoint applications typically need the Bursting Master to enable the host to provide information for the other modules.


Section Content
Tags

Level Two Title