R-tile Avalon® Streaming Intel® FPGA IP for PCI Express* User Guide

Download PDF
ID 683501
Date 6/20/2022
Public

A newer version of this document is available. Customers should click here to go to the newest version.

Document Table of Contents
Document Table of Contents x
1. About the R-tile Avalon® Streaming Intel® FPGA IP for PCI Express 2. IP Architecture and Functional Description 3. Advanced Features 4. Interfaces 5. Parameters 6. Troubleshooting/Debugging 7. R-Tile Avalon® Streaming Intel® FPGA IP for PCI Express* User Guide Archives 8. Document Revision History for the R-tile Avalon® Streaming Intel FPGA IP for PCI Express User Guide A. Configuration Space Registers B. Root Port Enumeration C. Implementation of Address Translation Services (ATS) in Endpoint Mode D. Packets Forwarded to the User Application in TLP Bypass Mode
1. About the R-tile Avalon® Streaming Intel® FPGA IP for PCI Express x
1.1. Overview 1.2. Features 1.3. Release Information 1.4. Device Family Support 1.5. Performance and Resource Utilization 1.6. IP Core Support Levels
1.1. Overview x
1.1.1. Standards and Specifications Compliance
1.2. Features x
1.2.1. Multifunction and Virtualization Features
2. IP Architecture and Functional Description x
2.1. Overview 2.2. Bias Temperature Instability (BTI) Protection Mode 2.3. PCIe Hard IP Mode 2.4. PIPE Direct Mode
2.3. PCIe Hard IP Mode x
2.3.1. Clocking 2.3.2. Reset 2.3.3. PCI Express Protocol Stack
2.3.2. Reset x
2.3.2.1. Single PERST 2.3.2.2. Independent PERST
2.3.2.2. Independent PERST x
2.3.2.2.1. REFCLK and PERST Guidelines when Using Independent PERST
2.3.3. PCI Express Protocol Stack x
2.3.3.1. PMA/PCS 2.3.3.2. Data Link Layer Overview 2.3.3.3. Transaction Layer Overview
2.3.3.3. Transaction Layer Overview x
2.3.3.3.1. Deferrable Memory Write (DMWr)
2.4. PIPE Direct Mode x
2.4.1. Clocking 2.4.2. Reset 2.4.3. PIPE Layer
3. Advanced Features x
3.1. PCIe Port Bifurcation and PHY Channel Mapping 3.2. Virtualization Support 3.3. TLP Bypass Mode
3.2. Virtualization Support x
3.2.1. SR-IOV Support 3.2.2. VirtIO Support
3.2.1. SR-IOV Support x
3.2.1.1. SR-IOV Supported Features List 3.2.1.2. Implementation 3.2.1.3. VF to PF Mapping 3.2.1.4. Function Level Reset (FLR)
3.2.1.2. Implementation x
3.2.1.2.1. VF Error Flag Interface (for x16/x8 Cores Only)
3.2.2. VirtIO Support x
3.2.2.1. VirtIO Supported Features List 3.2.2.2. Overview 3.2.2.3. Parameters 3.2.2.4. VirtIO PCI Configuration Access Interface 3.2.2.5. Registers
3.2.2.5. Registers x
3.2.2.5.1. VirtIO Common Configuration Capability Register (Address: 0x012) 3.2.2.5.2. VirtIO Common Configuration BAR Indicator Register (Address: 0x013) 3.2.2.5.3. VirtIO Common Configuration BAR Offset Register (Address: 0x014) 3.2.2.5.4. VirtIO Common Configuration Structure Length Register (Address 0x015) 3.2.2.5.5. VirtIO Notifications Capability Register (Address: 0x016) 3.2.2.5.6. VirtIO Notifications BAR Indicator Register (Address: 0x017) 3.2.2.5.7. VirtIO Notifications BAR Offset Register (Address: 0x018) 3.2.2.5.8. VirtIO Notifications Structure Length Register (Address: 0x019) 3.2.2.5.9. VirtIO Notifications Notify Off Multiplier Register (Address: 0x01A) 3.2.2.5.10. VirtIO ISR Status Capability Register (Address: 0x02F) 3.2.2.5.11. VirtIO ISR Status BAR Indicator Register (Address: 0x030) 3.2.2.5.12. VirtIO ISR Status BAR Offset Register (Address: 0x031) 3.2.2.5.13. VirtIO ISR Status Structure Length Register (Address: 0x032) 3.2.2.5.14. VirtIO Device Specific Capability Register (Address: 0x033) 3.2.2.5.15. VirtIO Device Specific BAR Indicator Register (Address: 0x034) 3.2.2.5.16. VirtIO Device Specific BAR Offset Register (Address 0x035) 3.2.2.5.17. VirtIO Device Specific Structure Length Register (Address: 0x036) 3.2.2.5.18. VirtIO PCI Configuration Access Capability Register (Address: 0x037) 3.2.2.5.19. VirtIO PCI Configuration Access BAR Indicator Register (Address: 0x038) 3.2.2.5.20. VirtIO PCI Configuration Access BAR Offset Register (Address: 0x039) 3.2.2.5.21. VirtIO PCI Configuration Access Structure Length Register (Address: 0x03A) 3.2.2.5.22. VirtIO PCI Configuration Access Data Register (Address: 0x03B)
3.3. TLP Bypass Mode x
3.3.1. Overview 3.3.2. Register Settings for the TLP Bypass Mode 3.3.3. Hard IP Reconfiguration Interface 3.3.4. Avalon® Streaming Interface
3.3.2. Register Settings for the TLP Bypass Mode x
3.3.2.1. TLPBYPASS_ERR_EN (Address 0x1314) 3.3.2.2. TLPBYPASS_ERR_STATUS (Address 0x1310)
3.3.4. Avalon® Streaming Interface x
3.3.4.1. Configuration TLP 3.3.4.2. Transmit Interface 3.3.4.3. Receive Interface 3.3.4.4. Malformed TLP 3.3.4.5. ECRC
4. Interfaces x
4.1. Clocks and Resets 4.2. Serial Data Interface 4.3. PCI Express Mode 4.4. PIPE Direct Mode
4.1. Clocks and Resets x
4.1.1. Clocks 4.1.2. Resets
4.3. PCI Express Mode x
4.3.1. Avalon® Streaming Interface 4.3.2. Precision Time Measurement (PTM) Interface (Endpoint Only) 4.3.3. Interrupt Interface 4.3.4. Hard IP Reconfiguration Interface 4.3.5. Error Interface 4.3.6. Completion Timeout Interface 4.3.7. Configuration Intercept Interface 4.3.8. Power Management Interface 4.3.9. Hard IP Status Interface 4.3.10. Page Request Services (PRS) Interface (Endpoint Only) 4.3.11. Function-Level Reset (FLR) Interface (Endpoint Only) 4.3.12. SR-IOV VF Error Flag Interface (Endpoint Only) 4.3.13. General Purpose VSEC Interface
4.3.1. Avalon® Streaming Interface x
4.3.1.1. TLP Header and Data Alignment for the Avalon® streaming RX and TX Interfaces 4.3.1.2. Credit Control 4.3.1.3. Avalon® Streaming RX Interface 4.3.1.4. Avalon® Streaming TX Interface 4.3.1.5. Tag Allocation
4.3.1.2. Credit Control x
4.3.1.2.1. Credit Interface - Data and Header 4.3.1.2.2. Credit Initialization 4.3.1.2.3. Credit Update/Release 4.3.1.2.4. RX Flow Control Interface 4.3.1.2.5. TX Flow Control Interface
4.3.1.4. Avalon® Streaming TX Interface x
4.3.1.4.1. Avalon® Streaming TX Interface tx_st_ready_o Behavior
4.3.1.5. Tag Allocation x
4.3.1.5.1. Completion Buffer Size
4.3.3. Interrupt Interface x
4.3.3.1. Legacy Interrupts 4.3.3.2. MSI 4.3.3.3. MSI-X
4.3.3.3. MSI-X x
4.3.3.3.1. Implementing MSI-X Interrupts
4.3.8. Power Management Interface x
4.3.8.1. D3Hot Entry 4.3.8.2. D3Hot Exit 4.3.8.3. D3Cold Entry 4.3.8.4. D3Cold Exit
4.3.8.2. D3Hot Exit x
4.3.8.2.1. D3Hot Exit Initiated by Host 4.3.8.2.2. D3Hot Exit Initiated by EP
4.4. PIPE Direct Mode x
4.4.1. Data Signals 4.4.2. Command and Status Signals 4.4.3. Message Bus Signals 4.4.4. Deskew Signals 4.4.5. PIPE Direct Reset Sequence 4.4.6. PIPE Direct Speed Change 4.4.7. PIPE Lane Reset Staggering 4.4.8. Unused Lanes in PIPE Direct Mode
4.4.1. Data Signals x
4.4.1.1. Transmit Signals 4.4.1.2. Receive Signals
4.4.4. Deskew Signals x
4.4.4.1. MAC to PHY (M2P) Signals 4.4.4.2. PHY to MAC (P2M) Signals
5. Parameters x
5.1. Top-Level Settings 5.2. Core Parameters
5.2. Core Parameters x
5.2.1. Avalon Parameters 5.2.2. Base Address Registers 5.2.3. PCI Express and PCI Capabilities Parameters
5.2.3. PCI Express and PCI Capabilities Parameters x
5.2.3.1. Device Capabilities 5.2.3.2. VirtIO Parameters 5.2.3.3. Link Capabilities 5.2.3.4. Legacy Interrupt Pin Register 5.2.3.5. MSI Capabilities 5.2.3.6. MSI-X Capabilities 5.2.3.7. Slot Capabilities 5.2.3.8. Latency Tolerance Reporting (LTR) 5.2.3.9. Process Address Space ID (PASID) 5.2.3.10. Device Serial Number Capability 5.2.3.11. Page Request Service (PRS) 5.2.3.12. Access Control Service (ACS) 5.2.3.13. Power Management 5.2.3.14. Vendor Specific Extended Capability (VSEC) Registers 5.2.3.15. TLP Processing Hints (TPH) 5.2.3.16. Address Translation Services (ATS) Capabilities 5.2.3.17. Precision Time Management (PTM)
6. Troubleshooting/Debugging x
6.1. Using the Hard IP Reconfiguration Interface for Debugging 6.2. Debug Toolkit
6.1. Using the Hard IP Reconfiguration Interface for Debugging x
6.1.1. Using the Hard IP Reconfiguration Interface to Enable and Read ECRC and LCRC Error Counts
6.2. Debug Toolkit x
6.2.1. Overview 6.2.2. Enabling the R-Tile Debug Toolkit 6.2.3. Launching the R-tile Debug Toolkit 6.2.4. Using the R-Tile Debug Toolkit
6.2.4. Using the R-Tile Debug Toolkit x
6.2.4.1. R-Tile Information 6.2.4.2. Event Counters 6.2.4.3. Configuration Space
C. Implementation of Address Translation Services (ATS) in Endpoint Mode x
C.1. Sending Translated/Untranslated Requests C.2. Sending a Page Request Message from the Endpoint (EP) to the Root Complex (RC) C.3. Invalidating Requests/Completions
D. Packets Forwarded to the User Application in TLP Bypass Mode x
D.1. EP TLP Bypass Mode (Upstream) D.2. RC TLP Bypass Mode (Downstream)
1. About the R-tile Avalon® Streaming Intel® FPGA IP for PCI Express
2. IP Architecture and Functional Description
3. Advanced Features
4. Interfaces
5. Parameters
6. Troubleshooting/Debugging
7. R-Tile Avalon® Streaming Intel® FPGA IP for PCI Express* User Guide Archives
8. Document Revision History for the R-tile Avalon® Streaming Intel FPGA IP for PCI Express User Guide
A. Configuration Space Registers
B. Root Port Enumeration
C. Implementation of Address Translation Services (ATS) in Endpoint Mode
D. Packets Forwarded to the User Application in TLP Bypass Mode

4.3.1.4. Avalon® Streaming TX Interface

The Application Layer transfers data to the Transaction Layer of the R-tile PCI Express IP core over the Avalon® -ST TX interface. The Transaction Layer must assert tx_st_ready_o before transmission begins. Transmission of a packet must be uninterrupted when tx_st_ready_o is asserted.

If this IP is configured in 1x16 with a double-width configuration, there are four segments with a 256-bit data width that allows multiple TLPs per cycle.

This interface supports one tx_st_sop_i signal and one tx_st_eop_i signal per cycle for each segment when the R-tile IP is operating in a 1x16 configuration. This means there are four tx_st_sop_i signals and four tx_st_eop_i signals for the x16 IP core. This interface also does not follow a fixed latency between the tx_st_ready_o and tx_st_valid_i signals as specified by the Avalon Interface Specifications. Data can be transmitted within the maximum latency between the deassertion of tx_st_ready_o and tx_st_valid_i, which is 16 coreclkout_hip cycles.

The x16 core in a double-width configuration provides four segments with each one having 256 bits of data (tx_st_data_i[255:0]), 128 bits of header (tx_st_hdr_i[127:0]), and 32 bits of TLP prefix (tx_st_tlp_prfx_i[31:0]). If this core is configured in the 1x16 mode, all four segments are used, so the data bus becomes a 1024-bit bus altogether, consisting of tx_st0_data_i[255:0], tx_st1_data_i[255:0], tx_st2_data_i[255:0], and tx_st3_data_1[255:0].

Parity generation is done via a 32:1 XOR (i.e. there is one parity bit for every 32 data, header or prefix bits).

Note: If the TLP length being transmitted is less than the segment size (255 bits), the corresponding tx_st_eop_i signal needs to happen in the same segment where tx_st_sop_i is being asserted. On the other hand, if the TLP length of the TLP being transmitted is greater than the segment size, the tx_st_eop_i signal can be asserted in the subsequent segments as dictated by the TLP length.
Note: tx_stN_sop_i pulses can only be sent on segments 0 or 2 (st0 or st2).
Table 55.  Avalon Streaming TX Interface Signals
Signal Name Direction Description EP/RP/BP Clock Domain
pX_tx_stN_data_i[255:0] where

X = 0,1,2,3 (IP core number)

N = 0,1,2,3 (segment number)

 Input

Application Layer data for transmission. The data bus is organized in multiple 256-bit segments. In x16 mode, all four segments are used to effectively form a 1024-bit data bus. In x8 mode, two segments are used to form a 512-bit data bus. In x4 mode, each 256-bit segment is an independent data bus.

The Application Layer must provide a properly formatted TLP on the TX interface. The data is valid when the corresponding tx_stN_valid_i signal is asserted.

The mapping of message TLPs is the same as the mapping of Transaction Layer TLPs with 4-dword headers. The number of data cycles must be correct for the length and address fields in the header. Issuing a packet with an incorrect number of data cycles results in the TX interface hanging and becoming unable to accept further requests.

Note: There must be no Idle cycle between the tx_stN_sop_i and tx_stN_eop_i cycles unless there is backpressure with the deassertion of tx_st_ready_o. 		EP/RP/BP coreclkout_hip
pX_tx_stN_hdr_i[127:0] where

X = 0,1,2,3 (IP core number)

N = 0,1,2,3 (segment number)

 Input This is the header to be transmitted, which follows the TLP header format of the PCIe specifications. Consider the following guidelines:
  • When the R-tile Avalon® Streaming IP for PCIe is configured in EP or RP mode, it automatically calculates the Completer/Requester ID and the Application logic does not need to provide this information as part of the TLP header being transmitted. Note that this guideline does not apply when the IP is configured in TLP Bypass mode.
  • When the R-tile Avalon® Streaming IP for PCIe is configured in EP mode and SR-IOV is enabled, follow the guidelines stated in the BDF Assignments section of SR-IOV Support Implementation.

These signals are valid when the corresponding tx_stN_sop_i signal is asserted.

EP/RP/BP coreclkout_hip
pX_tx_stN_prefix_i[31:0] where

X = 0,1,2,3 (IP core number)

N = 0,1,2,3 (segment number)

 Input

This is the TLP prefix to be transmitted, which follows the TLP prefix format of the PCIe specifications. PASID is supported.

These signals are valid when the corresponding tx_stN_sop_i signal is asserted.

The TLP prefix uses a Big Endian implementation (i.e. the Fmt field is in bits [31:29] and the Type field is in bits [28:24]).

If no prefix is present for a given TLP, that dword, including the Fmt field, is all zeros.

EP/RP/BP coreclkout_hip
pX_tx_stN_sop_i where

X = 0,1,2,3 (IP core number)

N = 0,2 (segment number)

 Input Indicate the first cycle of a TLP when asserted in conjunction with the corresponding bit of tx_stN_valid_i. For the x16 configuration:
  • tx_st3_sop_i: When asserted, indicates the start of a TLP in tx_st3_data_i[255:0].
  • tx_st2_sop_i: When asserted, indicates the start of a TLP in tx_st2_data_i[255:0].
  • tx_st1_sop_i: When asserted, indicates the start of a TLP in tx_st1_data_i[255:0].
  • tx_st0_sop_i: When asserted, indicates the start of a TLP in tx_st0_data_i[255:0].

These signals are asserted for one clock cycle per each TLP. They also qualify the corresponding tx_stN_hdr_i and tx_stN_tlp_prfx_i signals.

Note: pX_tx_stN_sop_i pulses can only be sent on segments 0 or 2 (st0 or st2).
EP/RP/BP coreclkout_hip
pX_tx_stN_eop_i where

X = 0,1,2,3 (IP core number)

N = 0,1,2,3 (segment number)

 Input Indicate the last cycle of a TLP when asserted in conjunction with the corresponding bit of tx_stN_valid_i. For the x16 configuration:
  • tx_st3_eop_i: When asserted, indicates the end of a TLP in tx_st3_data_i[255:0].
  • tx_st2_eop_i: When asserted, indicates the end of a TLP in tx_st2_data_i[255:0].
  • tx_st1_eop_i: When asserted, indicates the end of a TLP in tx_st1_data_i[255:0].
  • tx_st0_eop_i: When asserted, indicates the end of a TLP in tx_st0_data_i[255:0].

These signals are asserted for one clock cycle per each TLP.

EP/RP/BP coreclkout_hip
pX_tx_stN_dvalid_i where

X = 0,1,2,3 (IP core number)

N = 0,1,2,3 (segment number)

 Input

Qualify the data of the corresponding segment of tx_stN_data_i into the IP core on ready cycles.

To facilitate timing closure, Intel recommends that you register both the tx_st_ready_o and tx_stN_dvalid_i signals.

EP/RP/BP coreclkout_hip
pX_tx_stN_hvalid_i where

X = 0,1,2,3 (IP core number)

N = 0,1,2,3 (segment number)

 Input

Qualify the header of the corresponding segment of tx_stN_data_i into the IP core on ready cycles.

To facilitate timing closure, Intel recommends that you register both the tx_st_ready_o and tx_stN_hvalid_i signals.

EP/RP/BP coreclkout_hip
pX_tx_stN_pvalid_i where

X = 0,1,2,3 (IP core number)

N = 0,1,2,3 (segment number)

 Input

Qualify the prefix of the corresponding segment of tx_stN_data_i into the IP core on ready cycles.

To facilitate timing closure, Intel recommends that you register both the tx_st_ready_o and tx_stN_pvalid_i signals.

EP/RP/BP coreclkout_hip
pX_tx_stN_data_par_i[Z:0] where

X = 0,1,2,3 (IP core number) and Z varies based on the core.

N = 0,1,2,3 (segment number)

Input

Parity for tx_stN_data_i. Bit [0] corresponds to tx_stN_data_i[31:0], bit [1] corresponds to tx_stN_data_i[63:32], and so on.

By default, the PCIe Hard IP generates the parity for the TX data.

EP/RP/BP coreclkout_hip
pX_tx_stN_hdr_par_i[3:0] where

X = 0,1,2,3 (IP core number)

N = 0,1,2,3 (segment number)

 Input

Parity for tx_stN_hdr_i.

By default, the PCIe Hard IP generates the parity for the TX header.

EP/RP/BP coreclkout_hip
pX_tx_stN_prefix_par_i where

X = 0,1,2,3 (IP core number)

N = 0,1,2,3 (segment number)

 Input

Parity for tx_stN_tlp_prfx_i.

By default, the PCIe Hard IP generates the parity for the TX TLP prefix.

EP/RP/BP coreclkout_hip
pX_tx_st_ready_o where

X = 0,1,2,3 (IP core number)

Output

Indicates that the PCIe Hard IP is ready to accept data. The readyLatency maximum is 16 cycles.

If tx_st_ready_o is asserted by the Transaction Layer in the PCIe Hard IP on cycle <n>, then <n> + readyLatency is a ready cycle, during which the Application may assert tx_stN_valid_i and transfer data.

If tx_st_ready_o is deasserted by the Transaction Layer on cycle <n>, then the Application must deassert tx_stN_valid_i within the readyLatency number of cycles after cycle <n>.

tx_st_ready_o can be deasserted in the following conditions:
  • The LTSSM is not ready.
  • A Retry is in progress.
  • The R-tile Avalon-ST IP is busy sending internally generated TLPs.
  • The internal R-tile TX FIFO is full.
 EP/RP/BP coreclkout_hip

As an example, Avalon® Streaming TX Interface Timings below shows the behavior of the Avalon Streaming TX interface in a back-to-back TLPs scenario with data spanning across multiple segments. The following text describes the waveforms per clock cycle:

  1. Clock cycle 1: The R-tile Intel FPGA IP for PCI Express asserts p0_tx_st_ready_o signal, indicating the Hard IP is ready to accept TLPs from the Application logic.
  2. Clock cycle 2:
    1. The start of the first TLP (T0) is in segment 0, indicated by the assertion of p0_tx_st0_sop_i.
    2. The signal p0_tx_st0_hvalid_i is asserted to validate the header of this first TLP (T0H0) in the p0_tx_st0_hdr_i bus.
    3. The signal p0_tx_st0_dvalid_i is asserted to validate the data of this first TLP (T0D0) in the p0_tx_st0_data_i bus.
    4. The signal p0_tx_st1_dvalid_i is asserted to validate the next portion of the data of this first TLP (T0D1) in the p0_tx_st1_data_i bus.
    5. The signal p0_tx_st2_dvalid_i is asserted to validate the next portion of the data of this first TLP (T0D2) in the p0_tx_st2_data_i bus.
    6. The signal p0_tx_st3_dvalid_i is asserted to validate the final portion of the data of this first TLP (T0D3) in the p0_tx_st3_data_i bus.
    7. The end of this first TLP (T0) is in segment 3, denoted by the assertion of p0_tx_st3_eop_i.
  3. Clock cycle 3:
    1. The next TLP (T1), arrives in segment 0, as denoted by p0_tx_st0_sop_i staying high.
    2. The signal p0_tx_st0_hvalid_i is asserted to validate the header of this TLP (T1H0) in the p0_tx_st0_hdr_i bus.
    3. The signal p0_tx_st0_dvalid_i is asserted to validate the data of this TLP (T1D0) in the p0_tx_st0_data_i bus.
    4. The signal p0_tx_st1_dvalid_i is asserted to validate the next portion of the data of this TLP (T1D1) in the p0_tx_st1_data_i bus.
    5. The signal p0_tx_st2_dvalid_i is asserted to validate the next portion of the data of this TLP (T1D2) in the p0_tx_st2_data_i bus.
    6. The signal p0_tx_st3_dvalid_i is asserted to validate the final portion of the data of this TLP (T1D2) in the p0_tx_st3_data_i bus.
    7. The end of this TLP (T1) is in segment 3, denoted by p0_tx_st3_eop_i staying high.
Figure 30.  Avalon® Streaming TX Interface Timings
Note: tx_stN_sop_i pulses can only be sent on segments 0 or 2 (st0 or st2).

Section Content
Avalon Streaming TX Interface tx_st_ready_o Behavior

Level Two Title