F-Tile Avalon® Streaming Intel® FPGA IP for PCI Express* User Guide

Download PDF
ID 683140
Date 12/17/2021
Public

A newer version of this document is available. Customers should click here to go to the newest version.

Document Table of Contents
Document Table of Contents x
1. Acronyms 2. Introduction 3. IP Architecture and Functional Description 4. Advanced Features 5. Interfaces 6. Parameters 7. Testbench 8. Troubleshooting/Debugging 9. F-Tile Avalon Streaming Intel FPGA IP for PCI Express User Guide Archives 10. Revision History of the F-Tile Avalon Streaming Intel FPGA IP for PCI Express User Guide A. Configuration Space Registers B. Implementation of Address Translation Services (ATS) in Endpoint Mode C. Packets Forwarded to the User Application in TLP Bypass Mode D. Root Port Enumeration
2. Introduction x
2.1. Overview 2.2. Features 2.3. Release Information 2.4. Device Family Support 2.5. Resource Utilization 2.6. IP Core and Design Example Support Levels
3. IP Architecture and Functional Description x
3.1. Architecture 3.2. Functional Description 3.3. Avalon-ST TX/RX 3.4. Interrupts 3.5. Completion Timeout 3.6. Hot Plug 3.7. Power Management 3.8. Configuration Output Interface (COI) 3.9. Configuration Intercept Interface (EP Only) 3.10. Hard IP Reconfiguration Interface 3.11. PHY Reconfiguration Interface 3.12. Page Request Service (PRS) (EP Only)
3.1. Architecture x
3.1.1. Clocks 3.1.2. Refclk 3.1.3. Reset
3.2. Functional Description x
3.2.1. PMA/PCS 3.2.2. Data Link Layer 3.2.3. Transaction Layer
3.3. Avalon-ST TX/RX x
3.3.1. TLP Header and Data Alignment 3.3.2. Avalon-ST RX 3.3.3. Avalon-ST TX 3.3.4. Tag Allocation 3.3.5. Completion Buffer Size
3.3.2. Avalon-ST RX x
3.3.2.1. RX Flow Control
3.3.3. Avalon-ST TX x
3.3.3.1. TX Flow Control
3.4. Interrupts x
3.4.1. Legacy Interrupts 3.4.2. MSI 3.4.3. MSI-X
3.10. Hard IP Reconfiguration Interface x
3.10.1. Configuration Registers Access
3.10.1. Configuration Registers Access x
3.10.1.1. Direct User Avalon-MM Interface (Byte Access) 3.10.1.2. Debug Register Interface Access (Dword Access)
4. Advanced Features x
4.1. Virtualization Support 4.2. TLP Bypass Mode 4.3. Precision Time Measurement (PTM) 4.4. Scalable IOV
4.1. Virtualization Support x
4.1.1. Single Root I/O Virtualization (SR-IOV) 4.1.2. VirtIO
4.1.1. Single Root I/O Virtualization (SR-IOV) x
4.1.1.1. SR-IOV Implementation
4.1.1.1. SR-IOV Implementation x
4.1.1.1.1. Functional Level Reset (FLR)
4.1.2. VirtIO x
4.1.2.1. VirtIO Implementation 4.1.2.2. VirtIO Registers
4.2. TLP Bypass Mode x
4.2.1. Register Settings for TLP Bypass mode 4.2.2. Avalon-MM usage for TLP Bypass Mode 4.2.3. Transmit Interface 4.2.4. Receive Interface 4.2.5. Configuration TLP 4.2.6. Malformed TLP 4.2.7. ECRC
4.2.1. Register Settings for TLP Bypass mode x
4.2.1.1. TLPBYPASS_ERR_EN (Address 0x14194) 4.2.1.2. TLPBYPASS_ERR_STATUS (Address 0x14190)
5. Interfaces x
5.1. Overview 5.2. Clocks and Resets 5.3. Serial Data Interface 5.4. Avalon-ST Interface 5.5. Interrupt Interface 5.6. Hard IP Status Interface 5.7. Error Interface 5.8. 10-bit Tag Support Interface 5.9. Completion Timeout Interface 5.10. Power Management Interface 5.11. Hot Plug Interface (RP Only) 5.12. Configuration Output Interface 5.13. Configuration Intercept Interface (EP Only) 5.14. Hard IP Reconfiguration Interface 5.15. PHY Reconfiguration Interface 5.16. Page Request Service (PRS) Interface (EP Only) 5.17. FLR Interface Signals 5.18. PTM Interface Signals 5.19. VF Error Flag Interface Signals 5.20. VirtIO PCI Configuration Access Interface Signals
5.4. Avalon-ST Interface x
5.4.1. Avalon-ST RX Interface Signals 5.4.2. Avalon-ST TX Interface Signals
5.5. Interrupt Interface x
5.5.1. Legacy Interrupt Interface Signals 5.5.2. MSI Pending Bits Interface Signals
6. Parameters x
6.1. Top-Level Settings 6.2. Core Parameters
6.2. Core Parameters x
6.2.1. Avalon Parameters 6.2.2. Base Address Registers 6.2.3. PCI Express and PCI Capabilities Parameters 6.2.4. Device Identification Registers 6.2.5. Configuration, Debug and Extension Options
6.2.3. PCI Express and PCI Capabilities Parameters x
6.2.3.1. Device Capabilities 6.2.3.2. Link Capabilities 6.2.3.3. Legacy Interrupt Pin Register 6.2.3.4. MSI Capabilities 6.2.3.5. MSI-X Capabilities 6.2.3.6. Slot Capabilities 6.2.3.7. Latency Tolerance Reporting (LTR) 6.2.3.8. Process Address Space ID (PASID) 6.2.3.9. Device Serial Number Capability 6.2.3.10. Page Request Service (PRS) 6.2.3.11. Access Control Service (ACS) Capabilities 6.2.3.12. Power Management 6.2.3.13. Vendor Specific Extended Capability (VSEC) Registers 6.2.3.14. Precision Time Measurement (PTM) 6.2.3.15. Address Translation Services (ATS) 6.2.3.16. TLP Processing Hints (TPH) 6.2.3.17. VirtIO Parameters
7. Testbench x
7.1. Generating Tile Files 7.2. Endpoint Testbench 7.3. Test Driver Module 7.4. Root Port BFM 7.5. BFM Procedures and Functions
7.4. Root Port BFM x
7.4.1. BFM Memory Map 7.4.2. Configuration Space Bus and Device Numbering 7.4.3. Configuration of Root Port and Endpoint 7.4.4. Issuing Read and Write Transactions to the Application Layer
7.5. BFM Procedures and Functions x
7.5.1. ebfm_barwr Procedure 7.5.2. ebfm_barwr_imm Procedure 7.5.3. ebfm_barrd_wait Procedure 7.5.4. ebfm_barrd_nowt Procedure 7.5.5. ebfm_cfgwr_imm_wait Procedure 7.5.6. ebfm_cfgwr_imm_nowt Procedure 7.5.7. ebfm_cfgrd_wait Procedure 7.5.8. ebfm_cfgrd_nowt Procedure 7.5.9. BFM Configuration Procedures 7.5.10. BFM Shared Memory Access Procedures 7.5.11. BFM Log and Message Procedures 7.5.12. Verilog HDL Formatting Functions
7.5.9. BFM Configuration Procedures x
7.5.9.1. ebfm_cfg_rp_ep Procedure 7.5.9.2. ebfm_cfg_decode_bar Procedure
7.5.10. BFM Shared Memory Access Procedures x
7.5.10.1. Shared Memory Constants 7.5.10.2. shmem_write Task 7.5.10.3. shmem_read Function 7.5.10.4. shmem_display Verilog HDL Function 7.5.10.5. shmem_fill Procedure 7.5.10.6. shmem_chk_ok Function
7.5.11. BFM Log and Message Procedures x
7.5.11.1. ebfm_display Verilog HDL Function 7.5.11.2. ebfm_log_stop_sim Verilog HDL Function 7.5.11.3. ebfm_log_set_suppressed_msg_mask Task 7.5.11.4. ebfm_log_set_stop_on_msg_mask Verilog HDL Task 7.5.11.5. ebfm_log_open Verilog HDL Function 7.5.11.6. ebfm_log_close Verilog HDL Function
7.5.12. Verilog HDL Formatting Functions x
7.5.12.1. himage1 7.5.12.2. himage2 7.5.12.3. himage4 7.5.12.4. himage8 7.5.12.5. himage16 7.5.12.6. dimage1 7.5.12.7. dimage2 7.5.12.8. dimage3 7.5.12.9. dimage4 7.5.12.10. dimage5 7.5.12.11. dimage6 7.5.12.12. dimage7
8. Troubleshooting/Debugging x
8.1. Hardware 8.2. Debug Toolkit
8.1. Hardware x
8.1.1. Debugging Link Training Issues 8.1.2. Debugging Data Transfer and Performance Issues
8.1.1. Debugging Link Training Issues x
8.1.1.1. Generic Tools and Utilities 8.1.1.2. SignalTapII Logic Analyzer 8.1.1.3. Additional Debug Tools
8.1.1.3. Additional Debug Tools x
8.1.1.3.1. Using the Hard IP Reconfiguration Interface 8.1.1.3.2. Enable and Read LCRC and ECRC Error Count
8.1.2. Debugging Data Transfer and Performance Issues x
8.1.2.1. Advanced Error Reporting (AER) 8.1.2.2. Second-Level Debug Tools
8.2. Debug Toolkit x
8.2.1. Overview 8.2.2. Enabling the F-Tile Debug Toolkit 8.2.3. Launching the F-Tile Debug Toolkit 8.2.4. Using the F-Tile Debug Toolkit 8.2.5. Using the F-Tile Link Inspector
8.2.4. Using the F-Tile Debug Toolkit x
8.2.4.1. Main View 8.2.4.2. Toolkit Parameters 8.2.4.3. Channel Parameters 8.2.4.4. Eye Viewer
8.2.4.2. Toolkit Parameters x
8.2.4.2.1. F-Tile Information 8.2.4.2.2. Event Counter 8.2.4.2.3. P0/P1 Configuration Space
8.2.4.3. Channel Parameters x
8.2.4.3.1. General PHY 8.2.4.3.2. TX Path 8.2.4.3.3. RX Path
A. Configuration Space Registers x
A.1. Configuration Space Registers A.2. Configuration Space Registers for Virtualization A.3. Intel-Defined VSEC Capability Registers
A.1. Configuration Space Registers x
A.1.1. Register Access Definitions A.1.2. PCIe Configuration Header Registers A.1.3. PCI Express Capability Structures A.1.4. Physical Layer 16.0 GT/s Extended Capability Structure A.1.5. MSI-X Registers
A.2. Configuration Space Registers for Virtualization x
A.2.1. SR-IOV Virtualization Extended Capabilities Registers Address Map A.2.2. PCIe Configuration Registers for Each Virtual Function
A.2.2. PCIe Configuration Registers for Each Virtual Function x
A.2.2.1. Alternative Routing ID (ARI) Capability Structure A.2.2.2. TLP Processing Hint (TPH) Capability Structure A.2.2.3. Address Translation Services (ATS) Capability Structure A.2.2.4. Access Control Services (ACS) Capability Structure
A.2.2.1. Alternative Routing ID (ARI) Capability Structure x
A.2.2.1.1. ARI Enhanced Capability Header Register (Offset 0x0) A.2.2.1.2. ARI Capability and Control Register (Offset 0x4)
A.2.2.2. TLP Processing Hint (TPH) Capability Structure x
A.2.2.2.1. TPH Requester Enhanced Capability Header (Offset 0x0) A.2.2.2.2. TPH Requester Capability Register (Offset 0x4) A.2.2.2.3. TPH Requester Control Register (Offset 0x8)
A.2.2.3. Address Translation Services (ATS) Capability Structure x
A.2.2.3.1. ATS Enhanced Capability Header (Offset 0x0) A.2.2.3.2. ATS Capability Register and ATS Control Register (Offset 0x4)
A.2.2.4. Access Control Services (ACS) Capability Structure x
A.2.2.4.1. ACS Extended Capability Header (Offset 0x0) A.2.2.4.2. ACS Capability Register (Offset 0x4) A.2.2.4.3. ACS Control Register (Offset 0x6) A.2.2.4.4. Egress Control Vector (Offset 0x8)
A.3. Intel-Defined VSEC Capability Registers x
A.3.1. Intel-Defined VSEC Capability Header (Offset 00h) A.3.2. Intel-Defined Vendor Specific Header (Offset 04h) A.3.3. Intel Marker (Offset 08h) A.3.4. JTAG Silicon ID (Offset 0x0C - 0x18) A.3.5. User Configurable Device and Board ID (Offset 0x1C - 0x1D) A.3.6. General Purpose Control and Status Register (Offset 0x30) A.3.7. Uncorrectable Internal Error Status Register (Offset 0x34) A.3.8. Uncorrectable Internal Error Mask Register (Offset 0x38) A.3.9. Correctable Internal Error Status Register (Offset 0x3C) A.3.10. Correctable Internal Error Mask Register (Offset 0x40)
B. Implementation of Address Translation Services (ATS) in Endpoint Mode x
B.1. Sending Translated/Untranslated Requests B.2. Sending a Page Request Message from the Endpoint (EP) to the Root Complex (RC) B.3. Invalidating Requests/Completions
C. Packets Forwarded to the User Application in TLP Bypass Mode x
C.1. EP TLP Bypass Mode (Upstream) C.2. RC TLP Bypass Mode (Downstream)
1. Acronyms
2. Introduction
3. IP Architecture and Functional Description
4. Advanced Features
5. Interfaces
6. Parameters
7. Testbench
8. Troubleshooting/Debugging
9. F-Tile Avalon Streaming Intel FPGA IP for PCI Express User Guide Archives
10. Revision History of the F-Tile Avalon Streaming Intel FPGA IP for PCI Express User Guide
A. Configuration Space Registers
B. Implementation of Address Translation Services (ATS) in Endpoint Mode
C. Packets Forwarded to the User Application in TLP Bypass Mode
D. Root Port Enumeration

2.2. Features

The F-Tile Avalon® Streaming Interface for PCI Express supports the following features:

PCIe* Features

  • Complete protocol stack including the Transaction, Data Link and Physical Layers implemented as a Hard IP.
  • Topologies supported:
    Table 2.  Topologies Supported
      Gen3/Gen4 x16 Gen3/Gen4 x8 Gen3/Gen4 x4
    Endpoint

    Yes

    Yes

    Yes
    Root Port

    Yes

    Yes

    Yes
    TLP-Bypass

    Yes

    Yes

    Yes
    Note: Gen1/Gen2 or lower link width configurations are supported via link down-training
  • Supports up to 512-byte maximum payload size (MPS).
  • Supports Single Virtual Channel (VC).
  • Supports Completion Timeout Ranges through Completion Timeout Interface
  • Atomic Operations (FetchAdd/Swap/CAS).
  • Extended Tag Support. (10-bit Tag Support Applies to x16 ports only. Maximum 512 Outstanding Non-Posted Request)
  • Separate Refclk with Independent Spread Spectrum Clocking (SRIS).
  • Separate Refclk with no Spread Spectrum Clocking (SRNS).
  • Common Refclk architecture.
  • PCI Express* Advanced Error Reporting (Physical Function only).
  • ECRC generation and checking.
  • Data bus parity protection.
  • Supports D0 and D3 PCIe power states.
  • Lane Margining at Receiver.
  • Retimers presence detection.

Multifunction and Virtualization Features:

  • Single Root-IO Virtualization (SRIOV). Up to 2048 Virtual Functions.
  • ACS Control Service (ACS) capability support for Port 0 and 1 (x16 Core and x8 Core)
  • Alternative Routing-ID Interpretation (ARI).
  • Function Level Reset (FLR).
  • TLP Processing Hint (TPH).
    Note: TPH supports the "No Steering Tag (ST)" mode only.
  • Address Translation Services (ATS)
  • Supports Page Request Services (PRS).
  • Process Address Space ID (PASID).
  • Configuration Intercept Interface (for VirtIO).

Avalon® Streaming Interface IP Features:

  • User packet interface with separate header, data and prefix.
  • User packet interface with a split-bus architecture where the header, data and prefix busses consist of two segments each (x16 mode only). This improves the bandwidth efficiency of this interface as it can handle up to 2 TLPs in any given cycle.
  • Up to 512 outstanding Non-Posted request for Port 0 (x16 core)
  • Up to 256 outstanding Non-Posted request for Port 1,2 and 3 (x8 and x4 cores)
  • Supports Autonomous Hard IP mode. This mode allows the PCIe Hard IP to communicate with the Host before the FPGA configuration and entry into user mode are complete.
    Note: Unless Readiness Notifications mechanisms are used, the Root Complex and/or system software must allow at least one second after a Conventional Reset of a device before it may determine that a device that fails to return a Successful Completion status for a valid Configuration Request is a broken device. This period is independent of how quickly Link training is complete.
  • FPGA core configuration via PCIe link (CvP Init and CvP Update) supported by the x16Core or Port 0 only
  • Variable PLD clock frequencies: (350 MHz / 400MHz / 500 MHz for Intel® Agilex™ )
  • Legacy Interrupts
  • MSI/MSI-X interrupts
  • Configuration Extension Bus & VSEC via CII interface
    Note: CII Interface is not supported for 1 x4 configuration or Topology H.
  • Precision Time Measurement (PTM) (PTM Requester only)
    Note: You can enable PTM only in one core (either x16 or x8) at any given time.
  • Parity Support on Avalon-ST interface
  • The FPGA pin allocations for the F-Tile Avalon Streaming IP for PCI Express in the Intel® Quartus® Prime project is fixed. However, this IP does support lane reversal and polarity inversion on the PCB by default.
  • Modelsim and VCS are the simulators supported in the Intel® Quartus® Prime 21.3 release. Other simulators may be supported in a future release.

Standards and Specification Compliance

  • PCI Express Base Specification Revision 4.0
  • Single Root I/O Virtualization and Sharing Specification, Rev 1.1
  • Address Translation Services, Revision 1.1
  • PHY Interface for PCI Express Architectures, Version 4.x (the spec that corresponds to PCI Express Base Spec, Revision 4.0)
  • Virtual I/O Device (VIRTIO) Version 1.0
Note: Throughout this User Guide, the term AVST or Avalon-ST may be used as an abbreviation for the Avalon® Streaming Interface IP.
Level Two Title