F-Tile Avalon® Streaming Intel® FPGA IP for PCI Express* User Guide

Download PDF
ID 683140
Date 12/17/2021
Public

A newer version of this document is available. Customers should click here to go to the newest version.

Document Table of Contents
Document Table of Contents x
1. Acronyms 2. Introduction 3. IP Architecture and Functional Description 4. Advanced Features 5. Interfaces 6. Parameters 7. Testbench 8. Troubleshooting/Debugging 9. F-Tile Avalon Streaming Intel FPGA IP for PCI Express User Guide Archives 10. Revision History of the F-Tile Avalon Streaming Intel FPGA IP for PCI Express User Guide A. Configuration Space Registers B. Implementation of Address Translation Services (ATS) in Endpoint Mode C. Packets Forwarded to the User Application in TLP Bypass Mode D. Root Port Enumeration
2. Introduction x
2.1. Overview 2.2. Features 2.3. Release Information 2.4. Device Family Support 2.5. Resource Utilization 2.6. IP Core and Design Example Support Levels
3. IP Architecture and Functional Description x
3.1. Architecture 3.2. Functional Description 3.3. Avalon-ST TX/RX 3.4. Interrupts 3.5. Completion Timeout 3.6. Hot Plug 3.7. Power Management 3.8. Configuration Output Interface (COI) 3.9. Configuration Intercept Interface (EP Only) 3.10. Hard IP Reconfiguration Interface 3.11. PHY Reconfiguration Interface 3.12. Page Request Service (PRS) (EP Only)
3.1. Architecture x
3.1.1. Clocks 3.1.2. Refclk 3.1.3. Reset
3.2. Functional Description x
3.2.1. PMA/PCS 3.2.2. Data Link Layer 3.2.3. Transaction Layer
3.3. Avalon-ST TX/RX x
3.3.1. TLP Header and Data Alignment 3.3.2. Avalon-ST RX 3.3.3. Avalon-ST TX 3.3.4. Tag Allocation 3.3.5. Completion Buffer Size
3.3.2. Avalon-ST RX x
3.3.2.1. RX Flow Control
3.3.3. Avalon-ST TX x
3.3.3.1. TX Flow Control
3.4. Interrupts x
3.4.1. Legacy Interrupts 3.4.2. MSI 3.4.3. MSI-X
3.10. Hard IP Reconfiguration Interface x
3.10.1. Configuration Registers Access
3.10.1. Configuration Registers Access x
3.10.1.1. Direct User Avalon-MM Interface (Byte Access) 3.10.1.2. Debug Register Interface Access (Dword Access)
4. Advanced Features x
4.1. Virtualization Support 4.2. TLP Bypass Mode 4.3. Precision Time Measurement (PTM) 4.4. Scalable IOV
4.1. Virtualization Support x
4.1.1. Single Root I/O Virtualization (SR-IOV) 4.1.2. VirtIO
4.1.1. Single Root I/O Virtualization (SR-IOV) x
4.1.1.1. SR-IOV Implementation
4.1.1.1. SR-IOV Implementation x
4.1.1.1.1. Functional Level Reset (FLR)
4.1.2. VirtIO x
4.1.2.1. VirtIO Implementation 4.1.2.2. VirtIO Registers
4.2. TLP Bypass Mode x
4.2.1. Register Settings for TLP Bypass mode 4.2.2. Avalon-MM usage for TLP Bypass Mode 4.2.3. Transmit Interface 4.2.4. Receive Interface 4.2.5. Configuration TLP 4.2.6. Malformed TLP 4.2.7. ECRC
4.2.1. Register Settings for TLP Bypass mode x
4.2.1.1. TLPBYPASS_ERR_EN (Address 0x14194) 4.2.1.2. TLPBYPASS_ERR_STATUS (Address 0x14190)
5. Interfaces x
5.1. Overview 5.2. Clocks and Resets 5.3. Serial Data Interface 5.4. Avalon-ST Interface 5.5. Interrupt Interface 5.6. Hard IP Status Interface 5.7. Error Interface 5.8. 10-bit Tag Support Interface 5.9. Completion Timeout Interface 5.10. Power Management Interface 5.11. Hot Plug Interface (RP Only) 5.12. Configuration Output Interface 5.13. Configuration Intercept Interface (EP Only) 5.14. Hard IP Reconfiguration Interface 5.15. PHY Reconfiguration Interface 5.16. Page Request Service (PRS) Interface (EP Only) 5.17. FLR Interface Signals 5.18. PTM Interface Signals 5.19. VF Error Flag Interface Signals 5.20. VirtIO PCI Configuration Access Interface Signals
5.4. Avalon-ST Interface x
5.4.1. Avalon-ST RX Interface Signals 5.4.2. Avalon-ST TX Interface Signals
5.5. Interrupt Interface x
5.5.1. Legacy Interrupt Interface Signals 5.5.2. MSI Pending Bits Interface Signals
6. Parameters x
6.1. Top-Level Settings 6.2. Core Parameters
6.2. Core Parameters x
6.2.1. Avalon Parameters 6.2.2. Base Address Registers 6.2.3. PCI Express and PCI Capabilities Parameters 6.2.4. Device Identification Registers 6.2.5. Configuration, Debug and Extension Options
6.2.3. PCI Express and PCI Capabilities Parameters x
6.2.3.1. Device Capabilities 6.2.3.2. Link Capabilities 6.2.3.3. Legacy Interrupt Pin Register 6.2.3.4. MSI Capabilities 6.2.3.5. MSI-X Capabilities 6.2.3.6. Slot Capabilities 6.2.3.7. Latency Tolerance Reporting (LTR) 6.2.3.8. Process Address Space ID (PASID) 6.2.3.9. Device Serial Number Capability 6.2.3.10. Page Request Service (PRS) 6.2.3.11. Access Control Service (ACS) Capabilities 6.2.3.12. Power Management 6.2.3.13. Vendor Specific Extended Capability (VSEC) Registers 6.2.3.14. Precision Time Measurement (PTM) 6.2.3.15. Address Translation Services (ATS) 6.2.3.16. TLP Processing Hints (TPH) 6.2.3.17. VirtIO Parameters
7. Testbench x
7.1. Generating Tile Files 7.2. Endpoint Testbench 7.3. Test Driver Module 7.4. Root Port BFM 7.5. BFM Procedures and Functions
7.4. Root Port BFM x
7.4.1. BFM Memory Map 7.4.2. Configuration Space Bus and Device Numbering 7.4.3. Configuration of Root Port and Endpoint 7.4.4. Issuing Read and Write Transactions to the Application Layer
7.5. BFM Procedures and Functions x
7.5.1. ebfm_barwr Procedure 7.5.2. ebfm_barwr_imm Procedure 7.5.3. ebfm_barrd_wait Procedure 7.5.4. ebfm_barrd_nowt Procedure 7.5.5. ebfm_cfgwr_imm_wait Procedure 7.5.6. ebfm_cfgwr_imm_nowt Procedure 7.5.7. ebfm_cfgrd_wait Procedure 7.5.8. ebfm_cfgrd_nowt Procedure 7.5.9. BFM Configuration Procedures 7.5.10. BFM Shared Memory Access Procedures 7.5.11. BFM Log and Message Procedures 7.5.12. Verilog HDL Formatting Functions
7.5.9. BFM Configuration Procedures x
7.5.9.1. ebfm_cfg_rp_ep Procedure 7.5.9.2. ebfm_cfg_decode_bar Procedure
7.5.10. BFM Shared Memory Access Procedures x
7.5.10.1. Shared Memory Constants 7.5.10.2. shmem_write Task 7.5.10.3. shmem_read Function 7.5.10.4. shmem_display Verilog HDL Function 7.5.10.5. shmem_fill Procedure 7.5.10.6. shmem_chk_ok Function
7.5.11. BFM Log and Message Procedures x
7.5.11.1. ebfm_display Verilog HDL Function 7.5.11.2. ebfm_log_stop_sim Verilog HDL Function 7.5.11.3. ebfm_log_set_suppressed_msg_mask Task 7.5.11.4. ebfm_log_set_stop_on_msg_mask Verilog HDL Task 7.5.11.5. ebfm_log_open Verilog HDL Function 7.5.11.6. ebfm_log_close Verilog HDL Function
7.5.12. Verilog HDL Formatting Functions x
7.5.12.1. himage1 7.5.12.2. himage2 7.5.12.3. himage4 7.5.12.4. himage8 7.5.12.5. himage16 7.5.12.6. dimage1 7.5.12.7. dimage2 7.5.12.8. dimage3 7.5.12.9. dimage4 7.5.12.10. dimage5 7.5.12.11. dimage6 7.5.12.12. dimage7
8. Troubleshooting/Debugging x
8.1. Hardware 8.2. Debug Toolkit
8.1. Hardware x
8.1.1. Debugging Link Training Issues 8.1.2. Debugging Data Transfer and Performance Issues
8.1.1. Debugging Link Training Issues x
8.1.1.1. Generic Tools and Utilities 8.1.1.2. SignalTapII Logic Analyzer 8.1.1.3. Additional Debug Tools
8.1.1.3. Additional Debug Tools x
8.1.1.3.1. Using the Hard IP Reconfiguration Interface 8.1.1.3.2. Enable and Read LCRC and ECRC Error Count
8.1.2. Debugging Data Transfer and Performance Issues x
8.1.2.1. Advanced Error Reporting (AER) 8.1.2.2. Second-Level Debug Tools
8.2. Debug Toolkit x
8.2.1. Overview 8.2.2. Enabling the F-Tile Debug Toolkit 8.2.3. Launching the F-Tile Debug Toolkit 8.2.4. Using the F-Tile Debug Toolkit 8.2.5. Using the F-Tile Link Inspector
8.2.4. Using the F-Tile Debug Toolkit x
8.2.4.1. Main View 8.2.4.2. Toolkit Parameters 8.2.4.3. Channel Parameters 8.2.4.4. Eye Viewer
8.2.4.2. Toolkit Parameters x
8.2.4.2.1. F-Tile Information 8.2.4.2.2. Event Counter 8.2.4.2.3. P0/P1 Configuration Space
8.2.4.3. Channel Parameters x
8.2.4.3.1. General PHY 8.2.4.3.2. TX Path 8.2.4.3.3. RX Path
A. Configuration Space Registers x
A.1. Configuration Space Registers A.2. Configuration Space Registers for Virtualization A.3. Intel-Defined VSEC Capability Registers
A.1. Configuration Space Registers x
A.1.1. Register Access Definitions A.1.2. PCIe Configuration Header Registers A.1.3. PCI Express Capability Structures A.1.4. Physical Layer 16.0 GT/s Extended Capability Structure A.1.5. MSI-X Registers
A.2. Configuration Space Registers for Virtualization x
A.2.1. SR-IOV Virtualization Extended Capabilities Registers Address Map A.2.2. PCIe Configuration Registers for Each Virtual Function
A.2.2. PCIe Configuration Registers for Each Virtual Function x
A.2.2.1. Alternative Routing ID (ARI) Capability Structure A.2.2.2. TLP Processing Hint (TPH) Capability Structure A.2.2.3. Address Translation Services (ATS) Capability Structure A.2.2.4. Access Control Services (ACS) Capability Structure
A.2.2.1. Alternative Routing ID (ARI) Capability Structure x
A.2.2.1.1. ARI Enhanced Capability Header Register (Offset 0x0) A.2.2.1.2. ARI Capability and Control Register (Offset 0x4)
A.2.2.2. TLP Processing Hint (TPH) Capability Structure x
A.2.2.2.1. TPH Requester Enhanced Capability Header (Offset 0x0) A.2.2.2.2. TPH Requester Capability Register (Offset 0x4) A.2.2.2.3. TPH Requester Control Register (Offset 0x8)
A.2.2.3. Address Translation Services (ATS) Capability Structure x
A.2.2.3.1. ATS Enhanced Capability Header (Offset 0x0) A.2.2.3.2. ATS Capability Register and ATS Control Register (Offset 0x4)
A.2.2.4. Access Control Services (ACS) Capability Structure x
A.2.2.4.1. ACS Extended Capability Header (Offset 0x0) A.2.2.4.2. ACS Capability Register (Offset 0x4) A.2.2.4.3. ACS Control Register (Offset 0x6) A.2.2.4.4. Egress Control Vector (Offset 0x8)
A.3. Intel-Defined VSEC Capability Registers x
A.3.1. Intel-Defined VSEC Capability Header (Offset 00h) A.3.2. Intel-Defined Vendor Specific Header (Offset 04h) A.3.3. Intel Marker (Offset 08h) A.3.4. JTAG Silicon ID (Offset 0x0C - 0x18) A.3.5. User Configurable Device and Board ID (Offset 0x1C - 0x1D) A.3.6. General Purpose Control and Status Register (Offset 0x30) A.3.7. Uncorrectable Internal Error Status Register (Offset 0x34) A.3.8. Uncorrectable Internal Error Mask Register (Offset 0x38) A.3.9. Correctable Internal Error Status Register (Offset 0x3C) A.3.10. Correctable Internal Error Mask Register (Offset 0x40)
B. Implementation of Address Translation Services (ATS) in Endpoint Mode x
B.1. Sending Translated/Untranslated Requests B.2. Sending a Page Request Message from the Endpoint (EP) to the Root Complex (RC) B.3. Invalidating Requests/Completions
C. Packets Forwarded to the User Application in TLP Bypass Mode x
C.1. EP TLP Bypass Mode (Upstream) C.2. RC TLP Bypass Mode (Downstream)
1. Acronyms
2. Introduction
3. IP Architecture and Functional Description
4. Advanced Features
5. Interfaces
6. Parameters
7. Testbench
8. Troubleshooting/Debugging
9. F-Tile Avalon Streaming Intel FPGA IP for PCI Express User Guide Archives
10. Revision History of the F-Tile Avalon Streaming Intel FPGA IP for PCI Express User Guide
A. Configuration Space Registers
B. Implementation of Address Translation Services (ATS) in Endpoint Mode
C. Packets Forwarded to the User Application in TLP Bypass Mode
D. Root Port Enumeration

4.1.2.2. VirtIO Registers

The following VirtIO capability structure registers references apply to each PF and VF. Addresses shown are register addresses.

Table 27.  PF/VF Capability Link List when VirtIO is enabled
Capability Start Byte Address Last Byte Address DW Count
Type0 0x00 0x3F 16
PM (PF only) 0x40 0x47 2
VirtIO Common Configuration 0x48 0x57 4
VirtIO Notifications 0x58 0x6B 5
Reserved 0x6C 0x6F 1
PCIe 0x70 0xA3 13
Reserved 0xA4 0xAF 3
MSIX 0xB0 0xBB 3
VirtIO ISR Status 0xBC 0xCB 4
VirtIO Device-Specific Configuration 0xCC 0xDB 4
VirtIO PCI Configuration Access 0xDC 0xEF 5
Reserved 0xF0 0xFF 4
Table 28.  VirtIO Capability Structures
Address Name Description
VirtIO Common Configuration Capability Structure
012 Common Configuration Capability Register Capability ID, next capability pointer, capability length
013 BAR Indicator Register Lower 8 bits indicate which BAR holds the structure
014 BAR Offset Register Indicates starting address of the structure within the BAR
015 Structure Length Register Indicates length of structure
VirtIO Notifications Capability Structure
016 Notifications Capability Register Capability ID, next capability pointer, capability length
017 BAR Indicator Register Lower 8 bits indicate which BAR holds the structure
018 BAR Offset Register Indicates starting address of the structure within the BAR
019 Structure Length Register Indicates length of structure
01A Notify Off Multiplier Multiplier for queue_notify_off
VirtIO ISR Status Capability Structure
02F ISR Status Capability Register Capability ID, next capability pointer, capability length
030 BAR Indicator Register Lower 8 bits indicate which BAR holds the structure
031 BAR Offset Register Indicates starting address of the structure within the BAR
032 Structure Length Register Indicates length of structure
VirtIO Device-Specific Capability Structure (Optional)
033 Device Specific Capability Register Capability ID, next capability pointer, capability length
034 BAR Indicator Register Lower 8 bits indicate which BAR holds the structure
035 BAR Offset Register Indicates starting address of the structure within the BAR
036 Structure Length Register Indicates length of structure
VirtIO PCI Configuration Access Structure
037 PCI Configuration Access Capability Register Capability ID, next capability pointer, capability length
038 BAR Indicator Register Lower 8 bits indicate which BAR holds the structure
039 BAR Offset Register Indicates starting address of the structure within the BAR
03A Structure Length Register Indicates length of structure
03B PCI Configuration Data Data for BAR access
Table 29.  VirtIO Common Configuration Capability Register (Address: 0x012)
Bit Location Description Access Type Default Value
31:24 Configuration Type RO 0x01
23:16 Capability Length RO 0x10
15:8 Next Capability Pointer RO 0x58
7:0 Capability ID RO 0x09
Note: The capability register identifies that this is a vendor-specific capability. It also identifies the structure type.
Table 30.  VirtIO Common Configuration BAR Indicator Register (Address: 0x013)
Bit Location Description Access Type Default Value
31:24 Padding RO 0x00
23:16 Padding RO 0x00
15:8 Padding RO 0x00
7:0 BAR Indicator RO Programmable
Note: The Bar Indicator field holds the values 0x0 to 0x5 specifying a Base Address register (BAR) belonging to the function located beginning at 10h in PCI Configuration Space. The BAR is used to map the structure into the memory space. Any other value is reserved for future use.
Table 31.  VirtIO Common Configuration BAR Offset Register (Address: 0x014)
Bit Location Description Access Type Default Value
31:0 BAR Offset RO Programmable
Note: This register indicates where the structure begins relative to the base address associated with the BAR. The alignment requirements of the offset are indicated in each structure-specific section.
Table 32.  VirtIO Common Configuration Structure Length Register (Address: 0x015)
Bit Location Description Access Type Default Value
31:0 Structure Length RO Programmable
Note: The capability register identifies that this is a vendor-specific capability. It also identifies the structure type.
Table 33.  VirtIO Notifications Capability Register (Address: 0x016)
Bit Location Description Access Type Default Value
31:24 Configuration Type RO 0x02
23:16 Capability Length RO 0x14
15:8 Next Capability Pointer RO 0xBC
7:0 Capability ID RO 0x09
Note: The capability register identifies that this is a vendor-specific capability. It also identifies the structure type.
Table 34.  VirtIO Notifications BAR Indicator Register (Address: 0x017)
Bit Location Description Access Type Default Value
31:24 Padding RO 0x00
23:16 Padding RO 0x00
15:8 Padding RO 0x00
7:0 BAR Indicator RO Programmable
Note: The Bar Indicator field holds the values 0x0 to 0x5 specifying a Base Address register (BAR) belonging to the function located beginning at 10h in PCI Configuration Space. The BAR is used to map the structure into memory space. Any other value is reserved for future use.
Table 35.  VirtIO Notifications BAR Offset Register (Address: 0x018)
Bit Location Description Access Type Default Value
31:0 BAR Offset RO 31:0
Note: This register indicates where the structure begins relative to the base address associated with the BAR. The alignment requirements of the offset are indicated in each structure-specific section.
Table 36.  VirtIO Notifications Structure Length Register (Address: 0x019)
Bit Location Description Access Type Default Value
31:0 Structure Length RO 31:0
Note: The length register indicates the length of the structure. The length may include padding, fields unused by the driver, or future extensions.
Table 37.  VirtIO Notifications Notify Off Multiplier Register (Address: 0x01A)
Bit Location Description Access Type Default Value
31:0 Multiplier for queue_notify_off RO 31:0
Note: The notify off multiplier register indicates the multiplier for queue_notify_off in the structure.
Table 38.  VirtIO ISR Status Capability Register (Address: 0x02F)
Bit Location Description Access Type Default Value
31:24 Configuration Type RO 0x03
23:16 Capability Length RO 0x10
15:8 Next Capability Pointer RO If Device-Specific Capability is present, then points to 0xCC, else points to 0xDC.
7:0 Capability ID RO 0x09
Note: The capability register identifies that this is a vendor-specific capability. It also identifies the structure type.
Table 39.  VirtIO ISR Status BAR Indicator Register (Address: 0x030)
Bit Location Description Access Type Default Value
31:24 Padding RO 0x00
23:16 Padding RO 0x00
15:8 Padding RO 0x00
7:0 BAR Indicator RO Programmable
Note: The Bar Indicator field holds the values 0x0 to 0x5 specifying a Base Address register (BAR) belonging to the function located beginning at 10h in PCI Configuration Space. The BAR is used to map the structure into memory space. Any other value is reserved for future use.
Table 40.  VirtIO ISR Status BAR Offset Register (Address: 0x031)
Bit Location Description Access Type Default Value
31:0 BAR Offset RO Programmable
Note: This register indicates where the structure begins relative to the base address associated with the BAR. The alignment requirements of the offset are indicated in each structure-specific section.
Table 41.  VirtIO ISR Status Structure Length Register (Address: 0x032)
Bit Location Description Access Type Default Value
31:0 Structure Length RO Programmable
Note: The length register indicates the length of the structure. The length may include padding, fields unused by the driver, or future extensions.
Table 42.  VirtIO Device Specific Capability Register (Address: 0x033)
Bit Location Description Access Type Default Value
31:24 Configuration Type RO 0x04
23:16 Capability Length RO 0x10
15:8 Next Capability Pointer RO 0xDC
7:0 Capability ID RO 0x09
Note: The capability register identifies that this is a vendor-specific capability. It also identifies the structure type.
Table 43.  VirtIO Device Specific BAR Indicator Register (Address: 0x034)
Bit Location Description Access Type Default Value
31:24 Padding RO 0x00
23:16 Padding RO 0x00
15:8 Padding RO 0x00
7:0 BAR Indicator RO Programmable
Note: The BAR Indicator field holds the values 0x0 to 0x5 specifying a Base Address register (BAR) belonging to the function located beginning at 10h in PCI Configuration Space. The BAR is used to map the structure into memory space. Any other value is reserved for future use.
Table 44.  VirtIO Device Specific BAR Offset Register (Address: 0x035)
Bit Location Description Access Type Default Value
31:0 BAR Offset RO Programmable
Note: This register indicates where the structure begins relative to the base address associated with the BAR. The alignment requirements of the offset are indicated in each structure-specific section.
Table 45.  VirtIO Device Specific Structure Length Register (Address: 0x036)
Bit Location Description Access Type Default Value
31:0 Structure Length RO Programmable
Note: The length register indicates the length of the structure. The length may include padding, fields unused by the driver, or future extensions.
Table 46.  VirtIO PCI Configuration Access Capability Register (Address: 0x037)
Bit Location Description Access Type Default Value
31:24 Configuration Type RO 0x05
23:16 Capability Length RO 0x14
15:8 Next Capability Pointer RO 0x00
7:0 Capability ID RO 0x09
Note: The capability register identifies that this is a vendor-specific capability. It also identifies the structure type.
Table 47.  VirtIO PCI Configuration Access BAR Indicator Register (Address: 0x038)
Bit Location Description Access Type Default Value
31:24 Padding RO 0x00
23:16 Padding RO 0x00
15:8 Padding RO 0x00
7:0 BAR Indicator RWS Undetermined
Note: The BAR Indicator field holds the values 0x0 to 0x5 specifying a Base Address register (BAR) belonging to the function located beginning at 10h in PCI Configuration Space. The BAR is used to map the structure into memory space. Any other value is reserved for future use.
Note: Register attribute RWS means RW with sticky behavior. It doesn’t get reset by FLR, Hot Reset nor Warm Reset.
Table 48.  VirtIO PCI Configuration Access BAR Offset Register (Address: 0x039)
Bit Location Description Access Type Default Value
31:0 BAR Offset RWS Undetermined
Note: This register indicates where the structure begins relative to the base address associated with the BAR. The alignment requirements of the offset are indicated in each structure-specific section.
Note: Register attribute RWS means RW with sticky behavior. It doesn’t get reset by FLR, Hot Reset nor Warm Reset.
Table 49.  VirtIO PCI Configuration Access Structure Length Register (Address: 0x03A)
Bit Location Description Access Type Default Value
31:0 Structure Length RWS Undetermined
Note: The length register indicates the length of the structure. The length may include padding, fields unused by the driver, or future extensions.
Note: Register attribute RWS means RW with sticky behavior. It doesn’t get reset by FLR, Hot Reset nor Warm Reset.
Table 50.  VirtIO PCI Configuration Access Data Register (Address: 0x03B)
Bit Location Description Access Type Default Value
31:0 PCI Configuration Data RWS Undetermined
Note: The PCI configuration data register indicates the data for BAR access.
Note: Register attribute RWS means RW with sticky behavior. It doesn’t get reset by FLR, Hot Reset nor Warm Reset.
Level Two Title