Arria® 10 Avalon® Streaming with SR-IOV IP for PCIe* User Guide

Download PDF
ID 683686
Date 9/12/2024
Public
Document Table of Contents
Document Table of Contents x
1. Datasheet 2. Getting Started with the SR-IOV Design Example 3. Parameter Settings 4. Physical Layout 5. Interfaces and Signal Descriptions 6. Registers 7. Reset and Clocks 8. Programming and Testing SR-IOV Bridge MSI Interrupts 9. Error Handling 10. IP Core Architecture 11. Design Implementation 12. Debugging 13. Document Revision History A. Transaction Layer Packet (TLP) Header Formats B. Arria® 10 Avalon-ST with SR-IOV Interface for PCIe Solutions User Guide Archive
1. Datasheet x
1.1. Arria® 10 Avalon® -ST Interface with SR-IOV for PCI Express* Datasheet 1.2. Release Information 1.3. Device Family Support 1.4. Debug Features 1.5. IP Core Verification 1.6. Performance and Resource Utilization 1.7. Recommended Speed Grades for SR-IOV Interface
1.1. Arria® 10 Avalon® -ST Interface with SR-IOV for PCI Express* Datasheet x
1.1.1. SR-IOV Features
1.5. IP Core Verification x
1.5.1. Compatibility Testing Environment
2. Getting Started with the SR-IOV Design Example x
2.1. Directory Structure for Arria® 10 SR-IOV Design Example 2.2. Design Components for the SR-IOV Design Example 2.3. Generating the SR-IOV Design Example 2.4. Compiling and Simulating the Design for SR-IOV
3. Parameter Settings x
3.1. Parameters 3.2. Arria® 10 Avalon-ST Settings 3.3. Arria® 10 SR-IOV System Settings 3.4. Base Address Register (BAR) Settings 3.5. SR-IOV Device Identification Registers 3.6. Arria® 10 Interrupt Capabilities 3.7. Physical Function TLP Processing Hints (TPH) 3.8. Address Translation Services (ATS) 3.9. PCI Express and PCI Capabilities Parameters 3.10. PHY Characteristics 3.11. Example Designs
3.9. PCI Express and PCI Capabilities Parameters x
3.9.1. PCI Express and PCI Capabilities 3.9.2. Error Reporting 3.9.3. Link Capabilities 3.9.4. Slot Capabilities 3.9.5. Power Management
4. Physical Layout x
4.1. Hard IP Block Placement In Cyclone® 10 GX Devices 4.2. Hard IP Block Placement In Arria® 10 Devices 4.3. Channel and Pin Placement for the Gen1, Gen2, and Gen3 Data Rates 4.4. Channel Placement and fPLL and ATX PLL Usage for the Gen3 Data Rate 4.5. PCI Express Gen3 Bank Usage Restrictions
5. Interfaces and Signal Descriptions x
5.1. Avalon-ST TX Interface 5.2. Component-Specific Avalon-ST Interface Signals 5.3. Avalon-ST RX Interface 5.4. BAR Hit Signals 5.5. Configuration Status Interface 5.6. Clock Signals 5.7. Function-Level Reset (FLR) Interface 5.8. SR-IOV Interrupt Interface 5.9. Configuration Extension Bus (CEB) Interface 5.10. Implementing MSI-X Interrupts 5.11. Control Shadow Interface 5.12. Local Management Interface (LMI) Signals 5.13. Reset, Status, and Link Training Signals 5.14. Hard IP Reconfiguration Interface 5.15. Serial Data Signals 5.16. Test Signals 5.17. PIPE Interface Signals 5.18. Arria® 10 Development Kit Conduit Interface
5.9. Configuration Extension Bus (CEB) Interface x
5.9.1. Vital Product Data (VPD) Capability
5.9.1. Vital Product Data (VPD) Capability x
5.9.1.1. Determine the Pointer Address of an External Capability Register 5.9.1.2. VPD Capability Implementation
6. Registers x
6.1. Addresses for Physical and Virtual Functions 6.2. Correspondence between Configuration Space Registers and the PCIe Specification 6.3. PCI and PCI Express Configuration Space Registers 6.4. MSI Registers 6.5. MSI-X Capability Structure 6.6. Power Management Capability Structure 6.7. PCI Express Capability Structure 6.8. Advanced Error Reporting (AER) Enhanced Capability Header Register 6.9. Uncorrectable Error Status Register 6.10. Uncorrectable Error Mask Register 6.11. Uncorrectable Error Severity Register 6.12. Correctable Error Status Register 6.13. Correctable Error Mask Register 6.14. Advanced Error Capabilities and Control Register 6.15. Header Log Registers 0-3 6.16. SR-IOV Virtualization Extended Capabilities Registers 6.17. Virtual Function Registers
6.3. PCI and PCI Express Configuration Space Registers x
6.3.1. Type 0 Configuration Space Registers 6.3.2. PCI and PCI Express Configuration Space Register Content 6.3.3. Interrupt Line and Interrupt Pin Register
6.16. SR-IOV Virtualization Extended Capabilities Registers x
6.16.1. SR-IOV Virtualization Extended Capabilities Registers Address Map 6.16.2. ARI Enhanced Capability Header 6.16.3. SR-IOV Enhanced Capability Registers 6.16.4. Initial VFs and Total VFs Registers 6.16.5. VF Device ID Register 6.16.6. Page Size Registers 6.16.7. VF Base Address Registers (BARs) 0-5 6.16.8. Secondary PCI Express Extended Capability Header 6.16.9. Lane Status Registers 6.16.10. Transaction Processing Hints (TPH) Requester Enhanced Capability Header
7. Reset and Clocks x
7.1. Reset Sequence for Hard IP for PCI Express IP Core and Application Layer 7.2. Function Level Reset (FLR) 7.3. Clocks
7.3. Clocks x
7.3.1. Clock Domains 7.3.2. Clock Summary
7.3.1. Clock Domains x
7.3.1.1. coreclkout_hip 7.3.1.2. pld_clk
8. Programming and Testing SR-IOV Bridge MSI Interrupts x
8.1. Setting Up and Verifying MSI Interrupts 8.2. Masking MSI Interrupts 8.3. Dropping a Pending MSI Interrupt
9. Error Handling x
9.1. Physical Layer Errors 9.2. Data Link Layer Errors 9.3. Transaction Layer Errors 9.4. Error Reporting and Data Poisoning 9.5. Uncorrectable and Correctable Error Status Bits
10. IP Core Architecture x
10.1. PCI Express Protocol Stack 10.2. Data Link Layer 10.3. Physical Layer 10.4. Top-Level Interfaces 10.5. Arria® 10 Hard IP for PCI Express with Single-Root I/O Virtualization (SR-IOV)
10.4. Top-Level Interfaces x
10.4.1. Avalon-ST Interface 10.4.2. Clocks and Reset 10.4.3. Interrupts 10.4.4. PIPE
11. Design Implementation x
11.1. Making Pin Assignments to Assign I/O Standard to Serial Data Pins 11.2. Recommended Reset Sequence to Avoid Link Training Issues 11.3. SDC Timing Constraints
12. Debugging x
12.1. Setting Up Simulation 12.2. Simulation Fails To Progress Beyond Polling.Active State 12.3. Hardware Bring-Up Issues 12.4. Link Training 12.5. Creating a Signal Tap Debug File to Match Your Design Hierarchy 12.6. Use Third-Party PCIe Analyzer 12.7. BIOS Enumeration Issues
12.1. Setting Up Simulation x
12.1.1. Changing Between Serial and PIPE Simulation 12.1.2. Using the PIPE Interface for Gen1 and Gen2 Variants 12.1.3. Viewing the Important PIPE Interface Signals 12.1.4. Disabling the Scrambler for Gen1 and Gen2 Simulations 12.1.5. Disabling 8B/10B Encoding and Decoding for Gen1 and Gen2 Simulations
13. Document Revision History x
13.1. Document Revision History for the Arria® 10 Avalon® Streaming with SR-IOV IP for PCIe* User Guide
A. Transaction Layer Packet (TLP) Header Formats x
A.1. TLP Packet Formats without Data Payload A.2. TLP Packet Formats with Data Payload
1. Datasheet
2. Getting Started with the SR-IOV Design Example
3. Parameter Settings
4. Physical Layout
5. Interfaces and Signal Descriptions
6. Registers
7. Reset and Clocks
8. Programming and Testing SR-IOV Bridge MSI Interrupts
9. Error Handling
10. IP Core Architecture
11. Design Implementation
12. Debugging
13. Document Revision History
A. Transaction Layer Packet (TLP) Header Formats
B. Arria® 10 Avalon-ST with SR-IOV Interface for PCIe Solutions User Guide Archive

5.2. Component-Specific Avalon-ST Interface Signals

Table 23.  Component Specific Avalon- ST TX Signals

Signal

Direction

Description

Avalon-ST TX Physical and Virtual Function Identification Signals
tx_st_pf_num[1:0] Input

Identifies the Physical Function originating the TLP being transmitted on the TX Stream Interface. The user must provide the originating Function number on this input when transmitting memory requests, Completions, and messages routed by ID.

When the originating Function is a VF, this input must be set to the PF Number the VF is attached to.

This input is sampled by the SR-IOV bridge when tx_st_sop and tx_st_valid are both high.
tx_st_vf_active Input

The Application Layer must assert this input when transmitting a TLP driven by a Virtual Function on the TX Stream Interface.

The SR-IOV bridge samples this signal when tx_st_sop and tx_st_valid are both asserted.
tx_st_vf_num[10:0] Input

Identifies the Virtual Function driving the TLP being transmitted on the Avalon-ST TX interface. The Application Layer must provide the VF number offset of the originating Function on this input when transmitting memory requests, Completions, and messages routed by ID. Its value ranges from 0-<n>-1 where <n> is the number of VFs in the set of VFs attached to the associated PF. Up to 2048 VFs are supported.

The SR-IOV bridge samples this signal when when tx_st_sop, and tx_st_valid, and are asserted.

Not used when a PF is driving the TLP.
Avalon-ST TX Credit Signals
tx_cred_data_fc[11:0]

Output

Data credit limit for the received FC completions. Each credit is 16 bytes. There is a latency of two pld_clk clocks between a change on tx_cred_fc_sel and the corresponding data appearing on tx_cred_data_fc and tx_cred_hdr_fc.

tx_cred_fc_hip_cons[5:0]

Output

Asserted for 1 cycle each time the Hard IP consumes a credit. These credits are from messages that the Hard IP for PCIe generates for the following reasons:

  • To respond to memory read requests
  • To send error messages

This signal is not asserted when an Application Layer credit is consumed. For optimum performance the Application Layer can track of its own consumed credits. (The hard IP also tracks credits and deasserts tx_st_ready if it runs out of credits of any type.) To calculate the total credits consumed, the Application Layer can add its own credits consumed to those consumed by the Hard IP for PCIe. The credit signals are valid after the dlup (data link up) is asserted.

The 6 bits of this vector correspond to the following 6 types of credit types:

  • [5]: posted headers
  • [4]: posted data
  • [3]: non‑posted header
  • [2]: non‑posted data
  • [1]: completion header
  • [0]: completion data

During a single cycle, the IP core can consume either a single header credit or both a header and a data credit.

tx_cred_fc_infinite[5:0]

Output

When asserted, indicates that the corresponding credit type has infinite credits available and does not need to calculate credit limits. The 6 bits of this vector correspond to the following 6 types of credit types:

  • [5]: posted headers
  • [4]: posted data
  • [3]: non‑posted header
  • [2]: non‑posted data
  • [1]: completion header
  • [0]: completion data
tx_cred_fc_sel[1:0]

Input

Signal to select between the tx_cred_hdr_fc and tx_cred_data_fc outputs. There is a latency of two pld_clk clocks between a change on tx_cred_fc_sel and the corresponding data appearing on tx_cred_data_fc and tx_cred_hdr_fc. The following encoding are defined:

  • 2'b00: Output Posted credits
  • 2'b01: Output Non-Posted credits
  • 2'b10: Output Completions
tx_cred_hdr_fc[7:0]

Output

Header credit limit for the FC posted writes. Each credit is 20 bytes. There is a latency of two pld_clk clocks between a change on tx_cred_fc_sel and the corresponding data appearing on tx_cred_data_fc and tx_cred_hdr_fc.

tx_cons_cred_select Input

When 1, the output tx_cred_data* and tx_cred_hdr* signals specify the value of the hard IP internal credits-consumed counter. When 0, tx_cred_data* and tx_cred_hdr* signal specify the limit value.

This signal is present when you turn On Enable credit consumed selection port in the parameter editor .

The following table describes the signals that comprise the completion side band signals for the Avalon-ST interface. The Arria® 10 Hard IP for PCI Express provides a completion error interface that the Application Layer can use to report errors, such as programming model errors. When the Application Layer detects an error, it can assert the appropriate cpl_err bit to indicate what kind of error to log. If separate requests result in two errors, both are logged. The Hard IP sets the appropriate status bits for the errors in the Configuration Space. It also automatically sends error messages in accordance with the PCI Express Base Specification. Note that the Application Layer is responsible for sending the completion with the appropriate completion status value for non-posted requests. Refer to Error Handling for information on errors that are automatically detected and handled by the Hard IP.

For a description of the completion rules, the completion header format, and completion status field values, refer to Section 2.2.9 of the PCI Express Base Specification.

Table 24.  Completion Signals for the Avalon-ST Interface

Signal

Direction

Description

cpl_err[6:0]

Input

Completion error from a PF or VF This signal reports completion errors to the Configuration Space. The SR-IOV Bridge responds to the assertion of these bits by logging the status in the error reporting registers of the Function and sending error messages when required. When an error occurs, the appropriate signal is asserted for one cycle. The individual bits indicate following error or status conditions:

  • cpl_err[0]: Completion timeout error with recovery. This signal should be asserted when a master-like interface has performed a non-posted request that never receives a corresponding completion transaction after the 50 ms timeout period when the error is correctable. The SR-IOV Bridge sends a Correctable Error message to the Root Complex.
  • cpl_err[1]: Completion timeout error without recovery. This signal should be asserted when a master-like interface has performed a non-posted request that never receives a corresponding completion transaction after the 50 ms time-out period when the error is not correctable. .The SR-IOV Bridge sends a Fatal or Non-Fatal Error message to the Root Complex, The severity of the Completion Timeout error programmed in the AER Uncorrectable Error Severity Register determines the error message.
  • cpl_err[2]: Completer abort error. The Application Layer asserts this signal to respond to a non-posted request with a Completer Abort (CA) completion. The SR-IOV Bridge sends a Fatal or Correctable Error message to the Root Complex. The severity of the Completer Abort Sent error programmed in the AER Uncorrectable Error Severity Register of the Function determines the error message.
  • cpl_err[3]: Unexpected completion error. This signal must be asserted when an Application Layer master block detects an unexpected Completion.

    The SR-IOV Bridge sends a Fatal or Correctable Error message to the Root Complex. The severity of the Unexpected Completion Received error programmed in the Uncorrectable Error Severity Register of the Function determines the error message..

  • cpl_err[4]: Unsupported Request (UR) error for posted TLP. The Application Layer asserts this signal to treat a posted request as an Unsupported Request.

    The bridge also sends a Fatal or Non-Fatal Error message to the Root Complex, The severity of the error programmed in the AER Uncorrectable Error Severity Register of the Function determines the message.

  • cpl_err[5]: Unsupported Request error for non-posted TLP. The Application Layer asserts this signal to respond to a non-posted request with an Request (UR) completion.

    The bridge also sends a Fatal or Correctable Error message to the Root Complex, The severity of the error programmed in the AER Uncorrectable Error Severity Register of the Function determines the message.

  • cpl_err[6]: Log header. If header logging is required, this bit must be set in the every cycle in which any of cpl_err[2], cpl_err[3], cpl_err[4], or cpl_err[5]is set. The header must be supplied on the inputs log_hdr[127:0].
cpl_err_pf_num[<n>-1:0] Input Identifies the Function reporting the error oncpl_err inputs. When the Function is a VF, this input must specify the PF Number to which the VF is attached.
cpl_err_vf_active Input Indicates that the Function reporting the error is a Virtual Function. When this input is asserted, the VF number offset of the Function must be provided on cpl_err_vf_num[10:0].
cpl_err_vf_num[10:0] Input Whencpl_err_vf_active is asserted, this input identifies the VF number offset of the Function reporting the error. Its value ranges from 0-<n>-1, where <n>is the number of VFs in the set of VFs attached to the associated PF.
cpl_pending_pf[<n>-1:0] Input

Completion pending from the PF. The Application Layer must assert this signal when a PF has issued one of more Non-Posted transactions waiting for completions. For example, when a Non-Posted Request is pending from PF0. cpl_pending_pf[0] records pending completions for PF0. cpl_pending_pf[1] records pending completions for PF1.

log_hdr[127:0] Input When any of the bits 2, 3, 4, 5 of cpl_err is asserted, the Application Layer may provide the header of the TLP that caused the error condition. The order of bytes is the same as the order of the header bytes for the Avalon-ST streaming interfaces.
vf_compl_status_update Input Completion status update from the VF. The Application Layer must assert vf_compl_status_update whenever the Completion Pending status changes in a VF, The Application Layer must assert vf_compl_status_update until the SR-IOV bridge responds by setting vf_compl_status_update_ack.
vf_compl_status Input Must be set to the current Completion Pending status of the associated Function whenvf_compl_status_updateis asserted, The following encodings are defined:
  • 0: No Completion pending
  • 1: Completion pending for the Function.
vf_compl_status_pf_num[<n>-1:0] Input Must be set to the PF number associated with the signaling Function when vf_compl_status_updateis asserted, <n> is the number of PFs.
vf_compl_status_vf_num[<n>-1:0] Input Must be set to the VF offset associated with the signaling Function when vf_compl_status_update is asserted, <n> is the number of VFs.
vf_compl_status_update_ack Output The SR-IOV Bridge asserts vf_compl_status_update_ack for 1cycle when it has completed updating its internal state in response to vf_compl_status_update. The Application Layer must assert vf_compl_status_update high until vf_compl_status_update_ack is asserted.
Related Information
Level Two Title