R-tile Avalon® Streaming Intel® FPGA IP for PCI Express* User Guide

Download PDF
ID 683501
Date 12/13/2021
Public

A newer version of this document is available. Customers should click here to go to the newest version.

Document Table of Contents
Document Table of Contents x
1. Introduction 2. IP Architecture and Functional Description 3. Advanced Features 4. Interfaces 5. Parameters 6. R-tile Avalon® Streaming Intel® FPGA IP for PCI Express* User Guide Archives 7. Document Revision History for the R-tile Avalon® Streaming Intel FPGA IP for PCI Express User Guide A. Configuration Space Registers B. Root Port Enumeration C. Implementation of Address Translation Services (ATS) in Endpoint Mode D. Packets Forwarded to the User Application in TLP Bypass Mode
1. Introduction x
1.1. Overview 1.2. Features 1.3. Release Information 1.4. Device Family Support 1.5. Performance and Resource Utilization 1.6. IP Core Support Levels
1.1. Overview x
1.1.1. Standards and Specifications Compliance
2. IP Architecture and Functional Description x
2.1. Overview 2.2. Bias Temperature Instability (BTI) Protection Mode 2.3. PCIe Hard IP Mode 2.4. PIPE Direct Mode
2.3. PCIe Hard IP Mode x
2.3.1. Clocking 2.3.2. Reset 2.3.3. PCI Express Protocol Stack
2.3.3. PCI Express Protocol Stack x
2.3.3.1. PMA/PCS 2.3.3.2. Data Link Layer Overview 2.3.3.3. Transaction Layer Overview
2.4. PIPE Direct Mode x
2.4.1. Clocking 2.4.2. Reset 2.4.3. PIPE Layer
3. Advanced Features x
3.1. PCIe Port Bifurcation and PHY Channel Mapping 3.2. Virtualization Support 3.3. TLP Bypass Mode
3.2. Virtualization Support x
3.2.1. SR-IOV Support 3.2.2. VirtIO Support
3.2.1. SR-IOV Support x
3.2.1.1. SR-IOV Supported Features List 3.2.1.2. Implementation 3.2.1.3. VF to PF Mapping 3.2.1.4. Function Level Reset (FLR)
3.2.1.2. Implementation x
3.2.1.2.1. VF Error Flag Interface (for x16/x8 Cores Only)
3.2.2. VirtIO Support x
3.2.2.1. VirtIO Supported Features List 3.2.2.2. Overview 3.2.2.3. Parameters 3.2.2.4. VirtIO PCI Configuration Access Interface 3.2.2.5. Registers
3.2.2.5. Registers x
3.2.2.5.1. VirtIO Common Configuration Capability Register (Address: 0x012) 3.2.2.5.2. VirtIO Common Configuration BAR Indicator Register (Address: 0x013) 3.2.2.5.3. VirtIO Common Configuration BAR Offset Register (Address: 0x014) 3.2.2.5.4. VirtIO Common Configuration Structure Length Register (Address 0x015) 3.2.2.5.5. VirtIO Notifications Capability Register (Address: 0x016) 3.2.2.5.6. VirtIO Notifications BAR Indicator Register (Address: 0x017) 3.2.2.5.7. VirtIO Notifications BAR Offset Register (Address: 0x018) 3.2.2.5.8. VirtIO Notifications Structure Length Register (Address: 0x019) 3.2.2.5.9. VirtIO Notifications Notify Off Multiplier Register (Address: 0x01A) 3.2.2.5.10. VirtIO ISR Status Capability Register (Address: 0x02F) 3.2.2.5.11. VirtIO ISR Status BAR Indicator Register (Address: 0x030) 3.2.2.5.12. VirtIO ISR Status BAR Offset Register (Address: 0x031) 3.2.2.5.13. VirtIO ISR Status Structure Length Register (Address: 0x032) 3.2.2.5.14. VirtIO Device Specific Capability Register (Address: 0x033) 3.2.2.5.15. VirtIO Device Specific BAR Indicator Register (Address: 0x034) 3.2.2.5.16. VirtIO Device Specific BAR Offset Register (Address 0x035) 3.2.2.5.17. VirtIO Device Specific Structure Length Register (Address: 0x036) 3.2.2.5.18. VirtIO PCI Configuration Access Capability Register (Address: 0x037) 3.2.2.5.19. VirtIO PCI Configuration Access BAR Indicator Register (Address: 0x038) 3.2.2.5.20. VirtIO PCI Configuration Access BAR Offset Register (Address: 0x039) 3.2.2.5.21. VirtIO PCI Configuration Access Structure Length Register (Address: 0x03A) 3.2.2.5.22. VirtIO PCI Configuration Access Data Register (Address: 0x03B)
3.3. TLP Bypass Mode x
3.3.1. Overview 3.3.2. Register Settings for the TLP Bypass Mode 3.3.3. User Avalon Memory-mapped Interface 3.3.4. Avalon® Streaming Interface
3.3.2. Register Settings for the TLP Bypass Mode x
3.3.2.1. TLPBYPASS_ERR_EN (Address 0x1314) 3.3.2.2. TLPBYPASS_ERR_STATUS (Address 0x1310)
3.3.4. Avalon® Streaming Interface x
3.3.4.1. Configuration TLP 3.3.4.2. Transmit Interface 3.3.4.3. Receive Interface 3.3.4.4. Malformed TLP 3.3.4.5. ECRC
4. Interfaces x
4.1. Overview 4.2. Clocks and Resets 4.3. Serial Data Interface 4.4. PCI Express Mode 4.5. PIPE Direct Mode
4.2. Clocks and Resets x
4.2.1. Clocks 4.2.2. Resets
4.4. PCI Express Mode x
4.4.1. Avalon® Streaming Interface 4.4.2. Precision Time Measurement (PTM) Interface 4.4.3. Interrupt Interface 4.4.4. Hard IP Reconfiguration Interface 4.4.5. Error Interface 4.4.6. Completion Timeout Interface 4.4.7. Configuration Intercept Interface 4.4.8. Power Management Interface 4.4.9. Hard IP Status Interface 4.4.10. Page Request Services (PRS) Interface 4.4.11. Function-Level Reset (FLR) Interface 4.4.12. SR-IOV VF Error Flag Interface 4.4.13. General Purpose VSEC Interface
4.4.1. Avalon® Streaming Interface x
4.4.1.1. TLP Header and Data Alignment for the Avalon® streaming RX and TX Interfaces 4.4.1.2. Credit Control 4.4.1.3. Avalon® Streaming RX Interface 4.4.1.4. Avalon® Streaming TX Interface 4.4.1.5. Tag Allocation
4.4.1.2. Credit Control x
4.4.1.2.1. Credit Interface - Data and Header 4.4.1.2.2. Credit Initialization 4.4.1.2.3. Credit Update/Release 4.4.1.2.4. RX Flow Control Interface 4.4.1.2.5. TX Flow Control Interface
4.4.1.4. Avalon® Streaming TX Interface x
4.4.1.4.1. Avalon® Streaming TX Interface tx_st_ready_o Behavior
4.4.1.5. Tag Allocation x
4.4.1.5.1. Completion Buffer Size
4.4.3. Interrupt Interface x
4.4.3.1. Legacy Interrupts 4.4.3.2. MSI 4.4.3.3. MSI-X
4.4.3.3. MSI-X x
4.4.3.3.1. Implementing MSI-X Interrupts
4.4.8. Power Management Interface x
4.4.8.1. Endpoint D3Hot Entry 4.4.8.2. Endpoint D3Hot Exit
4.5. PIPE Direct Mode x
4.5.1. Data Signals 4.5.2. Message Channel 4.5.3. Deskew Channel 4.5.4. PIPE Direct Reset Sequence 4.5.5. PIPE Direct Speed Change 4.5.6. PIPE Lane Reset Staggering 4.5.7. Unused Lanes in PIPE Direct Mode 4.5.8. PIPE Direct Mode TX/RX Datapaths
4.5.1. Data Signals x
4.5.1.1. Transmit Signals 4.5.1.2. Receive Signals
5. Parameters x
5.1. Top-Level Settings 5.2. Core Parameters
5.2. Core Parameters x
5.2.1. Avalon Parameters 5.2.2. Base Address Registers 5.2.3. PCI Express and PCI Capabilities Parameters
5.2.3. PCI Express and PCI Capabilities Parameters x
5.2.3.1. Device Capabilities 5.2.3.2. VirtIO Parameters 5.2.3.3. Link Capabilities 5.2.3.4. Legacy Interrupt Pin Register 5.2.3.5. MSI Capabilities 5.2.3.6. MSI-X Capabilities 5.2.3.7. Slot Capabilities 5.2.3.8. Latency Tolerance Reporting (LTR) 5.2.3.9. Process Address Space ID (PASID) 5.2.3.10. Device Serial Number Capability 5.2.3.11. Page Request Service (PRS) 5.2.3.12. Access Control Service (ACS) 5.2.3.13. Power Management 5.2.3.14. Vendor Specific Extended Capability (VSEC) Registers 5.2.3.15. TLP Processing Hints (TPH) 5.2.3.16. Address Translation Services (ATS) Capabilities 5.2.3.17. Precision Time Management (PTM)
C. Implementation of Address Translation Services (ATS) in Endpoint Mode x
C.1. Sending Translated/Untranslated Requests C.2. Sending a Page Request Message from the Endpoint (EP) to the Root Complex (RC) C.3. Invalidating Requests/Completions
D. Packets Forwarded to the User Application in TLP Bypass Mode x
D.1. EP TLP Bypass Mode (Upstream) D.2. RC TLP Bypass Mode (Downstream)
1. Introduction
2. IP Architecture and Functional Description
3. Advanced Features
4. Interfaces
5. Parameters
6. R-tile Avalon® Streaming Intel® FPGA IP for PCI Express* User Guide Archives
7. Document Revision History for the R-tile Avalon® Streaming Intel FPGA IP for PCI Express User Guide
A. Configuration Space Registers
B. Root Port Enumeration
C. Implementation of Address Translation Services (ATS) in Endpoint Mode
D. Packets Forwarded to the User Application in TLP Bypass Mode

4.4.1.5.1. Completion Buffer Size

R-tile Hard IP for PCIe implements Completion (Cpl) buffers for header and data for each PCIe core. In Endpoint mode, when Completion credits are infinite, user application needs to manage the number of outstanding requests to prevent overflow and lost Completions.

Table 56.  Completion Buffer Size
Completion Buffer Depth Width
Port 0 Cpl header 572 N/A
Port 0 Cpl data 2048 256
Port 1 Cpl header 256 N/A
Port 1 Cpl data 1024 128
Port 2 Cpl header 128 N/A
Port 2 Cpl data 512 64
Port 3 Cpl header 128 N/A
Port 3 Cpl data 512 64

Application Layer should send a request when there are sufficient completion buffer entries. For a Memory Read request, the Completer is allowed to break up the response for a single Read Request into multiple Completions, so the Application Layer needs to account for the maximum possible Completions returned.

Below are a few examples for the amount of Completion buffer entries that would be required based on the Memory Read request size and the Read Completion Boundary (RCB) configuration.

Table 57.  Completion Buffer Entries Examples
Memory Read Request Examples Completion Buffer Entries Required
Port 0 Port 1 Port 2/3
Header Data Header Data Header Data
A Memory Read request with an address of 1_0000h and length of C0h bytes (192 decimal) can be completed by a Root Complex with an RCB value of 64 bytes with one of the following combinations of Completions (bytes):
  • 192
  • 128, 64
  • 64, 128
  • 64, 64, 64
3 6 3 12 3 24
A Memory Read request with an address of 1_0000h and length of C0h bytes (192 decimal) can be completed by a Root Complex with an RCB value of 128 bytes with one of the following combinations of Completions (bytes):
  • 192
  • 128, 64
2 6 2 12 2 24
A Memory Read request with an address of 1_0020h and length of 100h bytes (256 decimal) can be completed by a Root Complex with an RCB value of 64 bytes with one of the following combinations of Completions (bytes):
  • 256
  • 32, 224
  • 32, 64, 160
  • 32, 64, 64, 96
  • 32, 64, 64, 64, 32
  • 32, 64, 128, 32
  • 32, 128, 96
  • 32, 128, 64, 32
  • 96, 160
  • 96, 128, 32
  • 96, 64, 96
  • 96, 64, 64, 32
  • 160, 96
  • 160, 64, 32
  • 224, 32
 5 8 5 16 5 32

Based on the examples from the table above, Figure 28 below provides the suggested flow in the Application logic to track the completion buffer entries and based on this, schedule Non-Posted (NP) requests to the R-Tile Intel FPGA IP for PCI Express.

For illustration purposes, Figure 28 is showing the R-Tile Intel FPGA IP for PCIe express configured in Gen5x16 (Port 0), with the Application logic requesting a Memory Read of 192 Bytes and getting a single completion of 192 Bytes in the response from the link partner.

The flow is as follows:
  1. When dl_up signal goes from LOW to HIGH, the Application logic initializes the available completion buffer entries based on CPL buffer size table above (Table 56).
  2. When there is an NP request added to transmission queue of the Application logic, the Completion Tracking logic calculates the required completion buffer entries for the request, considering the maximum number of completions that could be received. For this example, it is 3 completions of 64 Bytes. (Refer to the table with the examples provided in Table 57).
  3. The Completion Tracking logic checks the required completion buffer entries against the available completion buffer entries.
  4. If there are sufficient buffer entries available, the Application logic schedules the Memory Read request for transmission.
    1. The Completion Tracking logic updates the available completion header buffer entries.
    2. The Completion Tracking logic updates the available completion data buffer entries.
  5. The R-Tile Intel FPGA IP for PCI Express sends the NP TLP to the link partner.
  6. The link partner responds to the MRd request with a single Completion with a data payload of 192B.
  7. When the Completion packet is received by the Application Layer in its Rx Queue,
    1. The Completion Tracking logic updates the available completion header buffer entries.
    2. The Completion Tracking logic updates the available completion data buffer entries.
  8. The Completion Tracking logic verifies if the original NP request has been fully completed with the Completion received.
    1. If the request has been completed, the Completion Tracking logic adjusts the available completion buffer entries based on the unused entries allocated to the request previously.
    2. In case of multiple Completions for steps 8 and 8a, the Completion Tracking Logic will update the completion buffer entries for each Completion TLP received accordingly.
Figure 28. Suggested Flow for Application Logic to Track Completion Buffer Entries and Schedule NP Requests to R-tile IP for PCIe

Level Two Title