F-Tile Avalon® Streaming Intel® FPGA IP for PCI Express* User Guide

Download PDF
ID 683140
Date 4/04/2022
Public

A newer version of this document is available. Customers should click here to go to the newest version.

Document Table of Contents
Document Table of Contents x
1. Acronyms 2. Introduction 3. IP Architecture and Functional Description 4. Advanced Features 5. Interfaces 6. Parameters 7. Testbench 8. Troubleshooting/Debugging 9. F-Tile Avalon Streaming Intel FPGA IP for PCI Express User Guide Archives 10. Revision History of the F-Tile Avalon Streaming Intel FPGA IP for PCI Express User Guide A. Configuration Space Registers B. Implementation of Address Translation Services (ATS) in Endpoint Mode C. Packets Forwarded to the User Application in TLP Bypass Mode D. Root Port Enumeration E. Bifurcated Endpoint Support for Independent Resets
2. Introduction x
2.1. Overview 2.2. Features 2.3. Release Information 2.4. Device Family Support 2.5. Resource Utilization 2.6. IP Core and Design Example Support Levels
3. IP Architecture and Functional Description x
3.1. Architecture 3.2. Functional Description 3.3. Avalon-ST TX/RX 3.4. Interrupts 3.5. Completion Timeout 3.6. Hot Plug 3.7. Power Management 3.8. Configuration Output Interface (COI) 3.9. Configuration Intercept Interface (EP Only) 3.10. Hard IP Reconfiguration Interface 3.11. PHY Reconfiguration Interface 3.12. Page Request Service (PRS) (EP Only)
3.1. Architecture x
3.1.1. Clocks 3.1.2. Refclk 3.1.3. Reset
3.2. Functional Description x
3.2.1. PMA/PCS 3.2.2. Data Link Layer 3.2.3. Transaction Layer
3.3. Avalon-ST TX/RX x
3.3.1. TLP Header and Data Alignment 3.3.2. Avalon-ST RX 3.3.3. Avalon-ST TX 3.3.4. Tag Allocation 3.3.5. Completion Buffer Size
3.3.2. Avalon-ST RX x
3.3.2.1. RX Flow Control
3.3.3. Avalon-ST TX x
3.3.3.1. TX Flow Control
3.4. Interrupts x
3.4.1. Legacy Interrupts 3.4.2. MSI 3.4.3. MSI-X
3.10. Hard IP Reconfiguration Interface x
3.10.1. Configuration Registers Access
3.10.1. Configuration Registers Access x
3.10.1.1. Direct User Avalon-MM Interface (Byte Access) 3.10.1.2. Debug Register Interface Access (Dword Access)
4. Advanced Features x
4.1. Virtualization Support 4.2. TLP Bypass Mode 4.3. Precision Time Measurement (PTM) 4.4. Scalable IOV
4.1. Virtualization Support x
4.1.1. Single Root I/O Virtualization (SR-IOV) 4.1.2. VirtIO
4.1.1. Single Root I/O Virtualization (SR-IOV) x
4.1.1.1. SR-IOV Implementation
4.1.1.1. SR-IOV Implementation x
4.1.1.1.1. Functional Level Reset (FLR)
4.1.2. VirtIO x
4.1.2.1. VirtIO Implementation 4.1.2.2. VirtIO Registers
4.2. TLP Bypass Mode x
4.2.1. Register Settings for TLP Bypass mode 4.2.2. Avalon-MM usage for TLP Bypass Mode 4.2.3. Transmit Interface 4.2.4. Receive Interface 4.2.5. Configuration TLP 4.2.6. Malformed TLP 4.2.7. ECRC
4.2.1. Register Settings for TLP Bypass mode x
4.2.1.1. TLPBYPASS_ERR_EN (Address 0x14194) 4.2.1.2. TLPBYPASS_ERR_STATUS (Address 0x14190)
5. Interfaces x
5.1. Overview 5.2. Clocks and Resets 5.3. Serial Data Interface 5.4. Avalon-ST Interface 5.5. Interrupt Interface 5.6. Hard IP Status Interface 5.7. Error Interface 5.8. 10-bit Tag Support Interface 5.9. Completion Timeout Interface 5.10. Power Management Interface 5.11. Hot Plug Interface (RP Only) 5.12. Configuration Output Interface 5.13. Configuration Intercept Interface (EP Only) 5.14. Hard IP Reconfiguration Interface 5.15. PHY Reconfiguration Interface 5.16. Page Request Service (PRS) Interface (EP Only) 5.17. FLR Interface Signals 5.18. PTM Interface Signals 5.19. VF Error Flag Interface Signals 5.20. VirtIO PCI Configuration Access Interface Signals
5.4. Avalon-ST Interface x
5.4.1. Avalon-ST RX Interface Signals 5.4.2. Avalon-ST TX Interface Signals
5.5. Interrupt Interface x
5.5.1. Legacy Interrupt Interface Signals 5.5.2. MSI Pending Bits Interface Signals
6. Parameters x
6.1. Top-Level Settings 6.2. Core Parameters
6.2. Core Parameters x
6.2.1. Avalon Parameters 6.2.2. Base Address Registers 6.2.3. PCI Express and PCI Capabilities Parameters 6.2.4. Device Identification Registers 6.2.5. Configuration, Debug and Extension Options
6.2.3. PCI Express and PCI Capabilities Parameters x
6.2.3.1. Device Capabilities 6.2.3.2. Link Capabilities 6.2.3.3. Legacy Interrupt Pin Register 6.2.3.4. MSI Capabilities 6.2.3.5. MSI-X Capabilities 6.2.3.6. Slot Capabilities 6.2.3.7. Latency Tolerance Reporting (LTR) 6.2.3.8. Process Address Space ID (PASID) 6.2.3.9. Device Serial Number Capability 6.2.3.10. Page Request Service (PRS) 6.2.3.11. Access Control Service (ACS) Capabilities 6.2.3.12. Power Management 6.2.3.13. Vendor Specific Extended Capability (VSEC) Registers 6.2.3.14. Precision Time Measurement (PTM) 6.2.3.15. Address Translation Services (ATS) 6.2.3.16. TLP Processing Hints (TPH) 6.2.3.17. VirtIO Parameters
7. Testbench x
7.1. Generating Tile Files 7.2. Endpoint Testbench 7.3. Test Driver Module 7.4. Root Port BFM 7.5. BFM Procedures and Functions
7.4. Root Port BFM x
7.4.1. BFM Memory Map 7.4.2. Configuration Space Bus and Device Numbering 7.4.3. Configuration of Root Port and Endpoint 7.4.4. Issuing Read and Write Transactions to the Application Layer
7.5. BFM Procedures and Functions x
7.5.1. ebfm_barwr Procedure 7.5.2. ebfm_barwr_imm Procedure 7.5.3. ebfm_barrd_wait Procedure 7.5.4. ebfm_barrd_nowt Procedure 7.5.5. ebfm_cfgwr_imm_wait Procedure 7.5.6. ebfm_cfgwr_imm_nowt Procedure 7.5.7. ebfm_cfgrd_wait Procedure 7.5.8. ebfm_cfgrd_nowt Procedure 7.5.9. BFM Configuration Procedures 7.5.10. BFM Shared Memory Access Procedures 7.5.11. BFM Log and Message Procedures 7.5.12. Verilog HDL Formatting Functions
7.5.9. BFM Configuration Procedures x
7.5.9.1. ebfm_cfg_rp_ep Procedure 7.5.9.2. ebfm_cfg_decode_bar Procedure
7.5.10. BFM Shared Memory Access Procedures x
7.5.10.1. Shared Memory Constants 7.5.10.2. shmem_write Task 7.5.10.3. shmem_read Function 7.5.10.4. shmem_display Verilog HDL Function 7.5.10.5. shmem_fill Procedure 7.5.10.6. shmem_chk_ok Function
7.5.11. BFM Log and Message Procedures x
7.5.11.1. ebfm_display Verilog HDL Function 7.5.11.2. ebfm_log_stop_sim Verilog HDL Function 7.5.11.3. ebfm_log_set_suppressed_msg_mask Task 7.5.11.4. ebfm_log_set_stop_on_msg_mask Verilog HDL Task 7.5.11.5. ebfm_log_open Verilog HDL Function 7.5.11.6. ebfm_log_close Verilog HDL Function
7.5.12. Verilog HDL Formatting Functions x
7.5.12.1. himage1 7.5.12.2. himage2 7.5.12.3. himage4 7.5.12.4. himage8 7.5.12.5. himage16 7.5.12.6. dimage1 7.5.12.7. dimage2 7.5.12.8. dimage3 7.5.12.9. dimage4 7.5.12.10. dimage5 7.5.12.11. dimage6 7.5.12.12. dimage7
8. Troubleshooting/Debugging x
8.1. Hardware 8.2. Debug Toolkit
8.1. Hardware x
8.1.1. Debugging Link Training Issues 8.1.2. Debugging Data Transfer and Performance Issues
8.1.1. Debugging Link Training Issues x
8.1.1.1. Generic Tools and Utilities 8.1.1.2. SignalTapII Logic Analyzer 8.1.1.3. Additional Debug Tools
8.1.1.3. Additional Debug Tools x
8.1.1.3.1. Using the Hard IP Reconfiguration Interface 8.1.1.3.2. Enable and Read LCRC and ECRC Error Count
8.1.2. Debugging Data Transfer and Performance Issues x
8.1.2.1. Advanced Error Reporting (AER) 8.1.2.2. Second-Level Debug Tools
8.2. Debug Toolkit x
8.2.1. Overview 8.2.2. Enabling the F-Tile Debug Toolkit 8.2.3. Launching the F-Tile Debug Toolkit 8.2.4. Using the F-Tile Debug Toolkit 8.2.5. Using the F-Tile Link Inspector
8.2.4. Using the F-Tile Debug Toolkit x
8.2.4.1. Main View 8.2.4.2. Toolkit Parameters 8.2.4.3. Channel Parameters 8.2.4.4. Eye Viewer
8.2.4.2. Toolkit Parameters x
8.2.4.2.1. F-Tile Information 8.2.4.2.2. Event Counter 8.2.4.2.3. P0/P1 Configuration Space
8.2.4.3. Channel Parameters x
8.2.4.3.1. General PHY 8.2.4.3.2. TX Path 8.2.4.3.3. RX Path
A. Configuration Space Registers x
A.1. Configuration Space Registers A.2. Configuration Space Registers for Virtualization A.3. Intel-Defined VSEC Capability Registers
A.1. Configuration Space Registers x
A.1.1. Register Access Definitions A.1.2. PCIe Configuration Header Registers A.1.3. PCI Express Capability Structures A.1.4. Physical Layer 16.0 GT/s Extended Capability Structure A.1.5. MSI-X Registers
A.2. Configuration Space Registers for Virtualization x
A.2.1. SR-IOV Virtualization Extended Capabilities Registers Address Map A.2.2. PCIe Configuration Registers for Each Virtual Function
A.2.2. PCIe Configuration Registers for Each Virtual Function x
A.2.2.1. Alternative Routing ID (ARI) Capability Structure A.2.2.2. TLP Processing Hint (TPH) Capability Structure A.2.2.3. Address Translation Services (ATS) Capability Structure A.2.2.4. Access Control Services (ACS) Capability Structure
A.2.2.1. Alternative Routing ID (ARI) Capability Structure x
A.2.2.1.1. ARI Enhanced Capability Header Register (Offset 0x0) A.2.2.1.2. ARI Capability and Control Register (Offset 0x4)
A.2.2.2. TLP Processing Hint (TPH) Capability Structure x
A.2.2.2.1. TPH Requester Enhanced Capability Header (Offset 0x0) A.2.2.2.2. TPH Requester Capability Register (Offset 0x4) A.2.2.2.3. TPH Requester Control Register (Offset 0x8)
A.2.2.3. Address Translation Services (ATS) Capability Structure x
A.2.2.3.1. ATS Enhanced Capability Header (Offset 0x0) A.2.2.3.2. ATS Capability Register and ATS Control Register (Offset 0x4)
A.2.2.4. Access Control Services (ACS) Capability Structure x
A.2.2.4.1. ACS Extended Capability Header (Offset 0x0) A.2.2.4.2. ACS Capability Register (Offset 0x4) A.2.2.4.3. ACS Control Register (Offset 0x6) A.2.2.4.4. Egress Control Vector (Offset 0x8)
A.3. Intel-Defined VSEC Capability Registers x
A.3.1. Intel-Defined VSEC Capability Header (Offset 00h) A.3.2. Intel-Defined Vendor Specific Header (Offset 04h) A.3.3. Intel Marker (Offset 08h) A.3.4. JTAG Silicon ID (Offset 0x0C - 0x18) A.3.5. User Configurable Device and Board ID (Offset 0x1C - 0x1D) A.3.6. General Purpose Control and Status Register (Offset 0x30) A.3.7. Uncorrectable Internal Error Status Register (Offset 0x34) A.3.8. Uncorrectable Internal Error Mask Register (Offset 0x38) A.3.9. Correctable Internal Error Status Register (Offset 0x3C) A.3.10. Correctable Internal Error Mask Register (Offset 0x40)
B. Implementation of Address Translation Services (ATS) in Endpoint Mode x
B.1. Sending Translated/Untranslated Requests B.2. Sending a Page Request Message from the Endpoint (EP) to the Root Complex (RC) B.3. Invalidating Requests/Completions
C. Packets Forwarded to the User Application in TLP Bypass Mode x
C.1. EP TLP Bypass Mode (Upstream) C.2. RC TLP Bypass Mode (Downstream)
E. Bifurcated Endpoint Support for Independent Resets x
E.1. PCI Express Resets E.2. Supported Configurations
E.2. Supported Configurations x
E.2.1. Configuration Type A E.2.2. Configuration Type B E.2.3. Configuration Type C
1. Acronyms
2. Introduction
3. IP Architecture and Functional Description
4. Advanced Features
5. Interfaces
6. Parameters
7. Testbench
8. Troubleshooting/Debugging
9. F-Tile Avalon Streaming Intel FPGA IP for PCI Express User Guide Archives
10. Revision History of the F-Tile Avalon Streaming Intel FPGA IP for PCI Express User Guide
A. Configuration Space Registers
B. Implementation of Address Translation Services (ATS) in Endpoint Mode
C. Packets Forwarded to the User Application in TLP Bypass Mode
D. Root Port Enumeration
E. Bifurcated Endpoint Support for Independent Resets

5.4.2. Avalon-ST TX Interface Signals

Table 63.  Avalon-ST TX Interface Signals
Signal Name Direction EP/RP/BP Clock Domain Description
p#_tx_st_data_i[w*128-1:0] Input EP/RP/BP coreclkout_hip

Application Layer data for transmission. The Application Layer must provide a properly formatted TLP on the TX interface. Valid when the corresponding tx_st_valid_i signal is asserted.

The mapping of message TLPs is the same as the mapping of Transaction Layer TLPs with 4-dword headers. The number of data cycles must be correct for the length and address fields in the header. Issuing a packet with an incorrect number of data cycles results in the TX interface hanging and becoming unable to accept further requests.

Note: There must be no Idle cycle between the tx_st_sop_i and tx_st_eop_i cycles unless there is backpressure with the deassertion of tx_st_ready_o.

p#_tx_st_sop_i[n-1:0] Input EP/RP/BP coreclkout_hip

Indicate the first cycle of a TLP when asserted in conjunction with the corresponding bit of tx_st_valid_i.

For the x16 configuration:
  • tx_st_sop_i[1]: When asserted, indicates the start of aTLP in tx_st_data_i[511:256].
  • tx_st_sop_i[0]: When asserted, indicates the start of a TLP in tx_st_data_i[255:0].

These signals are asserted for one clock cycle per each TLP. They also qualify the corresponding tx_st_hdr_i and tx_st_tlp_prfx_i signals.

p#_tx_st_eop_i[n-1:0] Input EP/RP/BP coreclkout_hip

Indicate the last cycle of a TLP when asserted in conjunction with the corresponding bit of tx_st_valid_i.

For the x16 configuration:
  • tx_st_eop_i[1]: When asserted, indicates the end of a TLP in tx_st_data_i[511:256].
  • tx_st_eop_i[0]: When asserted, indicates the end of a TLP in tx_st_data_i[255:0].

These signals are asserted for one clock cycle per each TLP.

p#_tx_st_valid_i[n-1:0] Input EP/RP/BP coreclkout_hip

Qualify the corresponding data segment of tx_st_data_i into the IP core on ready cycles.

To facilitate timing closure, it’s recommended that user register both the tx_st_ready_o and tx_st_valid_i signals.

Note: There must be no Idle cycle between the tx_st_sop_i and tx_st_eop_i cycles unless there is backpressure with the deassertion of tx_st_ready_o.

p#_tx_st_ready_o Output EP/RP/BP coreclkout_hip

Indicates that the PCIe Hard IP is ready to accept data for transmission. The readyLatency is three cycles.

If tx_st_ready_o is asserted by the Transaction Layer in the PCIe Hard IP on cycle <n>, then <n>+readyLatency is a ready cycle, during which the Application may assert tx_st_valid_i and transfer data.

If tx_st_ready_o is deasserted by the Transaction Layer on cycle <n>, then the Application must deassert tx_st_valid_i within the readyLatency number of cycles after cycle <n>.

tx_st_ready_o can be deasserted in the following conditions:
  • The LTSSM is not ready.
  • A Retry is in progress.
  • There are not enough credits available to send the request.
  • The F-Tile Avalon-ST IP is busy sending internally generated TLPs.
  • The internal F-Tile TX FIFO is full.
p#_tx_st_err_i[n-1:0] Input EP/RP/BP coreclkout_hip

When asserted, indicate an error in the transmitted TLP. These signals are asserted with tx_st_eop_i and nullify a packet.

tx_st_err_i[1]: When asserted, specifies an error in tx_st_data_i[511:256].

tx_st_err_i[0]: When asserted, specifies an error in tx_st_data_i[255:0].

p#_tx_st_hdr_i[n*128-1:0] Input EP/RP/BP coreclkout_hip
This is the header to be transmitted, which follows the TLP header format of the PCIe specifications except for the requester ID/completer ID fields (tx_st_hdr_i[95:80]):
  • tx_st_hdr_i[95:84]: tx_st_vf_num[11:0]
  • tx_st_hdr_i[83]: tx_st_vf_active
  • tx_st_hdr_i[82:80]: tx_st_func_num[2:0]

These signals are valid when the corresponding tx_st_sop_i signal is asserted.

The header uses a Big Endian implementation.

For TLP bypass, bit-127 is set to 1.

For EP/RP mode, bit-127 is set to 0.
p#_tx_st_tlp_prfx_i[n*32-1:0] Input EP/RP/BP coreclkout_hip

This is the TLP prefix to be transmitted, which follows the TLP prefix format of the PCIe specifications. PASID is included.

These signals are valid when the corresponding tx_st_sop_i signal is asserted.

The TLP prefix uses a Big Endian implementation (i.e. the Fmt field is in bits [31:29] and the Type field is in bits [28:24]).

If no prefix is present for a given TLP, that dword, including the Fmt field, is all zeros.

p#_tx_st_data_par_i[w*16-1:0] Input EP/RP/BP coreclkout_hip

Byte parity for tx_st_data_i. Bit[0] corresponds to tx_st_data_i[7:0], bit [1] corresponds to tx_st_data_i[15:8], and so on.

By default, the PCIe Hard IP generates the parity for the TX data.

However, when ECC is off, the parity can be passed in from the FPGA core by setting the k_pcie_parity_bypass register.

p#_tx_st_hdr_par_i[n*16-1:0] Input EP/RP/BP coreclkout_hip

Byte parity for tx_st_hdr_i. By default, the PCIe Hard IP generates the parity for the TX header.

However, when ECC is off, the parity can be passed in from the FPGA core by setting the k_pcie_parity_bypass register.

p#_tx_st_tlp_prfx_par_i[n*4-1:0] Input EP/RP/BP coreclkout_hip

Byte parity for tx_st_tlp_prfx_i. By default, the PCIe Hard IP generates the parity for the TX TLP prefix.

However, when ECC is off, the parity can be passed in from the FPGA core by setting the k_pcie_parity_bypass register.

p#_tx_par_err_o Output EP/RP/BP coreclkout_hip

Asserted for a single cycle to indicate a parity error during TX TLP transmission.

The IP core transmits TX TLP packets even when a parity error is detected.

Table 64.  TX Flow Control Interface Signals
Signal Name Direction EP/RP/BP Clock Domain Description
p#_tx_cdts_limit_tdm_idx_o[2:0] Output EP/RP/BP coreclkout_hip

Indicate the traffic type for the tx_cdts_limit_o[15:0] signals. This interface provides credit limit information for all enabled ports in a TDM manner.

The following encodings are defined:
  • 000: P header credit limit
  • 001: NP header credit limit
  • 010: CPL header credit limit
  • 011: reserved
  • 100: P data credit limit
  • 101: NP data credit limit
  • 110: CPL data credit limit
  • 111: reserved
p#_tx_cdts_limit_o[15:0] Output EP/RP/BP coreclkout_hip

Indicate the Flow Control (FC) credit units advertised by the remote Receiver.

These signals represent the total number of FC credits made available by the Receiver since Flow Control initialization.

Initially, these signals indicate the number of FC credits available in the remote Receiver. The value of these signals always increments and rolls over.

For example, if the remote Receiver advertises an initial Non-Posted Header (NPH) FC credit of 0xFFFF, after it receives a MRd request, the NPH FC credits value increments by 1 and rolls over to 0x0000. The tx_cdts_limit_tdm_idx_o[2:0]signals determine the traffic type.

When the traffic type is header credit, only the LSB 12 bits are valid.

Note that, in addition to the TLPs transmitted by the user application, internally generated TLPs also consume FC credits.

Level Two Title