P-Tile is an FPGA companion tile die that supports PCI Express* Gen4 in Endpoint, Root Port and TLP Bypass modes.

It serves as a companion tile for both Intel^® Stratix^® 10 DX and Intel^® Agilex™ devices.

P-Tile natively supports PCI Express* Gen3 and Gen4 configurations.

The P-tile Avalon^® streaming IP for PCI Express* supports the following features:

IP versions are the same as the Intel Quartus Prime Design Suite software versions up to v19.1. From Intel Quartus Prime Design Suite software version 19.2 or later, IPs have a new IP versioning scheme.

Intel verifies that the current version of the Intel^® Quartus^® Prime Pro Edition software compiles the previous version of each IP core, if this IP core was included in the previous release. Intel reports any exceptions to this verification in the Intel IP Release Notes or clarifies them in the Intel^® Quartus^® Prime Pro Edition IP Update tool. Intel does not verify compilation with IP core versions older than the previous release.

The following terms define device support levels for Intel^® FPGA IP cores:

• Advance support—the IP core is available for simulation and compilation for this device family. Timing models include initial engineering estimates of delays based on early post-layout information. The timing models are subject to change as silicon testing improves the correlation between the actual silicon and the timing models. You can use this IP core for system architecture and resource utilization studies, simulation, pinout, system latency assessments, basic timing assessments (pipeline budgeting), and I/O transfer strategy (data-path width, burst depth, I/O standards tradeoffs).
• Preliminary support—the IP core is verified with preliminary timing models for this device family. The IP core meets all functional requirements, but might still be undergoing timing analysis for the device family. It can be used in production designs with caution.
• Final support—the IP core is verified with final timing models for this device family. The IP core meets all functional and timing requirements for the device family and can be used in production designs.

The following table shows the typical resource utilization information for selected configurations.

The resource usage is based on the Avalon^® -ST IP core top-level entity (intel_pcie_ptile_ast) that includes IP core soft logic implemented in the FPGA fabric.

For details on the application clock frequencies that the IP core can support, refer to Table 7.

The four cores in the PCIe Hard IP can be configured to support the following topologies:

In Configuration Mode 0, only the x16 core is active, and it operates in x16 mode (in either Gen3 or Gen4).

In Configuration Mode 2, all four cores (x16, x8, x4_0, x4_1) are active, and they operate as four Gen3 x4 cores or four Gen4 x4 cores.

Each of the cores has its own Avalon^® -ST interface to the user logic. The number of IP-to-User Logic interfaces exposed to the FPGA fabric are different based on the configuration modes. For more details, refer to the Overview section of the Interfaces chapter.

The PHY clock domain (i.e. core_clk domain) is a dynamic frequency domain. The PHY clock frequency is dependent on the current link speed.

P-Tile has two reference clock inputs at the package level, refclk0 and refclk1. You must connect a 100 MHz reference clock source to these two inputs. Depending on the port mode, you can drive the two refclk inputs using either a single clock source or two independent clock sources.

In 1x16 and 4x4 modes, drive the refclk inputs with a single clock source (through a fanout buffer) as shown in the figure below.

The following is an example where a single PERST# (pin_perst_n) is driven with independent refclk0 and refclk1. In this example, the add-in card (FPGA and Soc) is powered up first. P-Tile refclk0 is fed by the on-board free-running oscillator. P-Tile refclk1 driven by the Host becomes stable later. Hence, the PERST# is connected to the Host.

Figure 5. Single PERST# Connection in Bifurcated 2x8 Mode

The P-Tile Avalon^® -ST IP for PCI Express contains Physical Medium Attachment (PMA) and PCI Express Physical Coding Sublayer (PCIe PCS) blocks for handling the Physical layer (PHY) packets. The PMA receives and transmits high-speed serial data on the serial lanes. The PCS acts as an interface between the PMA and the PCIe controller, and performs functions like data encoding and decoding, scrambling and descrambling, block synchronization etc. The PCIe PCS in the P-Tile Avalon^® -ST IP for PCI Express is based on the PHY Interface for PCI Express (PIPE) Base Specification 4.4.1.

In this IP, the PMA consists of up to four quads. Each quad contains a pair of transmit PLLs and four SerDes lanes capable of running up to 16 GT/s to perform the various TX and RX functions.

PLLA generates the required transmit clocks for Gen1/Gen2 speeds, while PLLB generates the required clocks for Gen3/Gen4 speeds. For the x8 and x16 lane widths, one of the quads acts as the master PLL source to drive the clock inputs for each of the lanes in the other quads.

The PMA performs functions such as serialization/deserialization, clock data recovery, and analog front-end functions such as Continuous Time Linear Equalizer (CTLE), Decision Feedback Equalizer (DFE) and transmit equalization.

The transmitter consists of a 3-tap equalizer with one tap of pre-cursor, one tap of main cursor and one tap of post-cursor.

The PHY layer uses a fixed 16-bit PCS-PMA interface width to output the PHY clock (core_clk). The frequency of this clock is dependent on the current link speed. Refer to Table 7 for the frequencies at various link speeds.

The Data Link Layer (DLL) is located between the Transaction Layer and the Physical Layer. It maintains packet integrity and communicates (by DLL packet transmission) at the PCI Express link level.

The DLL implements the following functions:

• Link management through the reception and transmission of DLL Packets (DLLP), which are used for the following functions:
  • Power management of DLLP reception and transmission
  • To transmit and receive ACK/NAK packets
  • Data integrity through the generation and checking of CRCs for TLPs and DLLPs
  • TLP retransmission in case of NAK DLLP reception or replay timeout, using the retry (replay) buffer
  • Management of the retry buffer
  • Link retraining requests in case of error through the Link Training and Status State Machine (LTSSM) of the Physical Layer

The DLL has the following sub-blocks:

• Data Link Control and Management State Machine—This state machine connects to both the Physical Layer’s LTSSM state machine and the Transaction Layer. It initializes the link and flow control credits and reports status to the Transaction Layer.
• Power Management—This function handles the handshake to enter low power mode. Such a transition is based on register values in the Configuration Space and received Power Management (PM) DLLPs. For more details on the power states supported by the P-Tile Avalon^® -ST IP for PCIe, refer to section Power Management Interface.
• Data Link Layer Packet Generator and Checker—This block is associated with the DLLP’s 16-bit CRC and maintains the integrity of transmitted packets.
• Transaction Layer Packet Generator—This block generates transmit packets, including a sequence number and a 32-bit Link CRC (LCRC). The packets are also sent to the retry buffer for internal storage. In retry mode, the TLP generator receives the packets from the retry buffer and generates the CRC for the transmit packet.
• Retry Buffer—The retry buffer stores TLPs and retransmits all unacknowledged packets in the case of NAK DLLP reception. In case of ACK DLLP reception, the retry buffer discards all acknowledged packets.
• ACK/NAK Packets—The ACK/NAK block handles ACK/NAK DLLPs and generates the sequence number of transmitted packets.
• Transaction Layer Packet Checker—This block checks the integrity of the received TLP and generates a request for transmission of an ACK/NAK DLLP.
• TX Arbitration—This block arbitrates transactions, prioritizing in the following order:
  • Initialize FC Data Link Layer packet
  • ACK/NAK DLLP (high priority)
  • Update FC DLLP (high priority)
  • PM DLLP
  • Retry buffer TLP
  • TLP
  • Update FC DLLP (low priority)
  • ACK/NAK FC DLLP (low priority)

The following figure shows the major blocks in the P-Tile Avalon^® -ST IP for PCI Express Transaction Layer:

The RAS (Reliability, Availability, and Serviceability) block includes a set of features to maintain the integrity of the link.

For example: Transaction Layer inserts an optional ECRC in the transmit logic and checks it in the receive logic to provide End-to-End data protection.

When the application logic sets the TLP Digest (TD) bit in the Header of the TLP, the P-Tile Avalon^® -ST IP for PCIe will append the ECRC automatically.

Note that in TLP Bypass mode, the PCIe Hard IP does not generate/check the ECRC and will not remove it if the received TLP has the ECRC.

The TX block sends out the TLPs that it receives as-is. It also sends the information about non-posted TLPs to the CPL Timeout Block for CPL timeout detection.

This chapter provides a reference for all the parameters that are configurable in the Intel^® Quartus^® Prime IP Parameter Editor for the P-Tile Avalon^® -ST IP for PCIe.

Depending on which Hard IP Mode you choose in the Top-Level Settings tab, you will see different tabs for setting the core parameters.

Figure 10. Intel P-tile Avalon^® -ST Top-Level IP Parameter Editor for a x16 Hard IP Mode If you choose a x16 mode (either Gen4 or Gen3), only the PCIe0 Settings tab will appear.

Figure 11. Enabling TLP Bypass Mode

Figure 12. Intel P-tile Avalon^® -ST Top-Level IP Parameter Editor for a x8 Hard IP Mode If you choose a x8 mode (either Gen4 or Gen3), the PCIe0 Settings and PCIe1 Settings tabs will appear.

Figure 13. Intel P-tile Avalon^® -ST Top-Level IP Parameter Editor for a x4 Hard IP Mode If you choose a x4 mode (either Gen4 or Gen3), the PCIe0 Settings, PCIe1 Settings, PCIe2 Settings and PCIe3 Settings tabs will appear.

The following table lists the default values of the Device ID registers. You can use the parameter editor to change the values of these registers.

For each core (PCIe0/PCIe1/PCIe2/PCIe3), the PCI Express / PCI Capabilities tab contains separate tabs for the device, PRS (Endpoint mode), MSI (Endpoint mode), ACS capabilities (Root Port mode), slot (Root Port mode), MSI-X, and legacy interrupt pin register parameters.

Figure 14. PCI Express / PCI Capabilities Parameters

This capability allows the P-Tile Avalon streaming IP, when operating in Endpoint mode, to report the delay that it can tolerate when requesting service from the Host. This information can help software optimize performance when the Endpoint needs a fast response, or optimize system power when a fast response is not necessary.

To enable VirtIO support, first enable the support for multiple physical functions in the IP Parameter Editor as shown in the following screenshot:

Figure 16. Enable Multifunction Support

Make sure that SR-IOV support is also enabled:

Figure 17. Enable SR-IOV Support

Enable VirtIO support as shown in the screenshot below:

Figure 18. Enable VirtIO Support

Finally, you can configure the appropriate VirtIO capability parameters in the tabs shown in the screenshot below:

Figure 19. Configure VirtIO Capability Parameters

The following table provides a reference for all the configurable high-level parameters of the VirtIO block for P-Tile. Parameters below are dedicated to each core.

The next table summarizes the parameters associated with the five VirtIO device configuration structures:

Figure 20 shows the top-level signals of this IP. Note that the signal names in the figure will get the appropriate prefix pn (where n = 0, 1, 2 or 3) depending on which of the three supported configurations (1x16, 2x8, or 4x4) the P-Tile Avalon^® -ST IP for PCI Express* is in.

The only cases where the interface signal names do not get the pn prefixes are the interfaces that are common for all the cores, like the PHY reconfiguration interface, clocks and resets. For example, there is only one xcvr_reconfig_clk that is shared by all the cores.

You can enable the PHY reconfiguration interface from the Top Level Settings in the GUI.

Each of the cores has its own Avalon^® -ST interface to the user logic. The number of IP-to-User Logic interfaces exposed to the FPGA fabric are different based on the configuration modes:

FLR allows specific physical/virtual functions to be reset without affecting other physical/virtual functions or the link they share. FLR can be enabled by checking the check-box Enable Function Level Reset (FLR) in the PCIe Device tab of the PCIe PCI Express / PCI Capabilities tab in the GUI.

This interface is only present in EP mode (for x16/x8 configurations).

P-Tile natively supports 4, 8, or 16 PCIe lanes. Each lane includes a TX differential pair and an RX differential pair. Data is striped across all available lanes.

The P-tile PCIe Hard IP provides an Avalon^® -ST-like interface with separate header and data to improve the bandwidth utilization.

The Avalon^® -ST interface has different data bus widths depending on the link width configuration of the PCIe IP.

The TLP prefix, header and data are sent and received on the TX and RX interfaces.

The ordering of bytes in the header and data portions of packets is different. The first byte of the header dword is located in the most significant byte of the dword. The first byte of the data dword is located in the least significant byte of the dword on the data bus.

Figure 21. Generic TLP Format

Figure 22. TLP Prefix, Header and Data on the RX and TX Interfaces of the P-Tile IP for PCIe

The Application Layer receives data from the Transaction Layer of the PCI Express* IP core over the Avalon^® -ST RX interface. The application must assert rx_st_ready_i before transfers can begin.

This interface supports two rx_st_sop_o signals and two rx_st_eop_o signals per cycle when the P-Tile IP is operating in a x16 configuration. It also does not follow a fixed latency between rx_st_ready_i and rx_st_valid_o as specified by the Avalon Interface Specifications.

The x16 core provides two segments with each one having 256 bits of data (rx_st_data_o[511:256] and rx_st_data_o[255:0]), 128 bits of header (rx_st_hdr_o[255:128] and rx_st_hdr_o[127:0]), and 32 bits of TLP prefix (rx_st_tlp_prfx_o[63:32] and rx_st_tlp_prfx_o[31:0]). If this core is configured in the 1x16 mode, both segments are used, so the data bus becomes a 512-bit bus rx_st_data_o[511:0]. The start of packet can appear in the upper segment or lower segment, as indicated by the rx_st_sop_o[1:0] signals.

If this core is configured in the 2x8 mode, only the lower segment is used. In this case, the data bus is a 256-bit bus rx_st_data_o[255:0].

Finally, if this core is configured in the 4x4 mode, only the lower segment is used and only the MSB 128 bits of data are valid. In this case, the data bus is a 128-bit bus rx_st_data_o[127:0].

The x8 core provides one segment with 256 bits of data, 128 bits of header and 32 bits of TLP prefix. If this core is configured in 4x4 mode, only the LSB 128 bits of data are used.

The x4 core provides one segment with 128 bits of data, 128 bits of header and 32 bits of TLP prefix.

The following timing diagram illustrates the timing of the RX interface when the application throttles the P-Tile IP for PCIe by deasserting rx_st_ready_i. The Transaction Layer in the P-Tile IP deasserts rx_st_valid_o within 27 cycles of the rx_st_ready_i deassertion. It also reasserts rx_st_valid_o within 27 cycles after rx_st_ready_i reasserts if there is more data to send. rx_st_data_o is held until the application is able to accept it.

The RX flow control interface provides information on the application's available RX buffer space to the PCIe Hard IP in a time-division multiplexing (TDM) manner. It reports the space available in number of TLPs.

The RX flow control interface is optional and disabled by default in the IP GUI. If disabled, it indicates that there is no limit in the application RX buffer space.

For more details on the usage of the scale factors, refer to Section 3.4.2 of the PCI Express Base Specification, Rev. 4.0 Version 1.0.

The Application Layer transfers data to the Transaction Layer of the PCI Express* IP core over the Avalon^® -ST TX interface. The Transaction Layer must assert tx_st_ready_o before transmission begins. Transmission of a packet must be uninterrupted when tx_st_ready_o is asserted.

This 512-bit interface supports two locations for the beginning of a TLP, bit[0] and bit[256]. The interface supports multiple TLPs per cycle only when an end-of-packet cycle occurs in the lower 256 bits.

The x16 core provides two segments with each one having 256 bits of data (tx_st_data_i[511:256] and tx_st_data_i[255:0]), 128 bits of header (tx_st_hdr_i[255:128] and tx_st_hdr_i[127:0]), and 32 bits of TLP prefix (tx_st_tlp_prfx_i[63:32] and tx_st_tlp_prfx_i[31:0]). If this core is configured in the 1x16 mode, both segments are used, so the data bus becomes a 512-bit bus tx_st_data_i[511:0]. The start of packet can appear in the upper segment or lower segment, as indicated by the tx_st_sop_i[1:0] signals.

If this core is configured in the 2x8 mode, only the lower segment is used. In this case, the data bus is a 256-bit bus tx_st_data_i[255:0].

Finally, if this core is configured in the 4x4 mode, only the lower segment is used and only the LSB 128 bits of data are valid. In this case, the data bus is a 128-bit bus tx_st_data_i[127:0].

The x8 core provides one segment with 256 bits of data, 128 bits of header and 32 bits of TLP prefix. If this core is configured in 4x4 mode, only the LSB 128 bits of data are used.

The x4 core provides one segment with 128 bits of data, 128 bits of header and 32 bits of TLP prefix.

The following timing diagram illustrates the behavior of tx_st_ready_o, which is deasserted to pause the data transmission to the Transaction Layer of the P-Tile IP for PCIe, and then reasserted. The timing diagram shows a readyLatency of three cycles. The application deasserts tx_st_valid_i three cycles after tx_st_ready_o is deasserted.

The application must not deassert tx_st_valid_i between tx_st_sop_i and tx_st_eop_i on a ready cycle. For the definition of a ready cycle, refer to the Avalon^® Interface Specifications.

Before a TLP can be transmitted, flow control logic verifies that the link partner's RX port has sufficient buffer space to accept it. The TX Flow Control interface reports the link partner's available RX buffer space to the Application. It reports the space available in units called Flow Control credits for posted, non-posted and completion TLPs (as defined in the RX Flow Control Interface section).

TX credit limit signals are provided in a TDM manner similar to how the RX credit limit signals are provided.

This example shows how this interface is updated when multiple MWr requests are sent. The tx_cdts_limit_o[15:0] bus value is incremented when a TLP is acknowledged by the receiver and will roll over when reaching 0xFFFF.

The P-Tile PCIe Hard IP supports the 10-bit tag Requester capability in the x16 Controller (Port 0) only. It supports up to 512 outstanding Non-Posted Requests (NPRs) with valid tag values ranging from 256 to 767.

The x8 (Port 1) and x4 Controllers (Port 2/3) don’t support the 10-bit tag Requester capability, although they support the 10-bit Completer capability.

Both x8 and x4 Controllers can allow up to 256 outstanding NPRs with valid tag values ranging from 0 to 255.

Note that all PFs and their associated VFs share the same tag space. This means that different PFs and VFs cannot have outstanding tags having the same tag values.

In the TLP bypass mode, there is no restriction on the tag allocation since the P-Tile PCIe Hard IP does not do any tag management. Hence, 10-bit tags can be used without any restriction across all the cores.

This interface includes the signals that are useful for debugging, such as the link status signal, LTSSM state outputs, etc. These signals are available when the optional Power Management interface is enabled.

The P-Tile Avalon^® -ST IP for PCI Express* supports Message Signaled Interrupts (MSI), MSI-X interrupts, and legacy interrupts. MSI and legacy interrupts are mutually exclusive.

The user application generates MSI which are single-Dword memory write TLPs to implement interrupts. This interrupt mechanism conserves pins because it does not use separate wires for interrupts. In addition, the single Dword provides flexibility for the data presented in the interrupt message. The MSI Capability structure is stored in the Configuration Space and is programmed using Configuration Space accesses.

The user application generates MSI-X messages which are single-Dword memory writes. The MSI-X Capability structure points to an MSI-X table structure and an MSI-X Pending Bit Array (PBA) structure which are stored in memory. This scheme is different than the MSI Capability structure, which contains all the control and status information for the interrupts.

Enable legacy interrupts by programming the Interrupt Disable bit (bit[10]) of the Configuration Space Command to 1'b0. When legacy interrupts are enabled, the IP core emulates INTx interrupts using virtual wires. The app_int_i ports control legacy interrupt generation.

Legacy interrupts mimic the original PCI level-sensitive interrupts using virtual wire messages. The P-tile IP for PCIe signals legacy interrupts on the PCIe link using Message TLPs. The term INTx refers collectively to the four legacy interrupts, INTA#,INTB#, INTC# and INTD#. The P-tile IP for PCIe asserts app_int_i to cause an Assert_INTx Message TLP to be generated and sent upstream. A deassertion of app_int_i, i.e a transition of this signal from high to low, causes a Deassert_INTx Message TLP to be generated and sent upstream. To use legacy interrupts, you must clear the Interrupt Disable bit, which is bit 10 of the Command Register in the configuration header. Then, you must turn off the MSI Enable bit.

app_int_i[0] is asserted for at least eight clock cycles to cause an Assert_INTx Message TLP to be generated and sent upstream for physical function 0. For a multi-functions implementation, app_int_i[0] is for physical function 0, app_int_i[1] is for physical function 1 and so on. Deasserting an app_int_i signal by driving it from high to low causes a Deassert_INTx Message TLP to be generated and sent upstream.

MSI interrupts are signaled on the PCI Express link using a single dword Memory Write TLP. The user application issues an MSI request (MWr) through the Avalon^® -ST interface and updates the configuration space register using the MSI interface.

For more details on the MSI Capability Structure, refer to Figure 84.

The Mask Bits register and Pending Bits register are 32 bits in length each, with each potential interrupt message having its own mask bit and pending bit. If bit[0] of the Mask Bits register is set, interrupt message 0 is masked. When an interrupt message is masked, the MSI for that vector cannot be sent. If software clears the mask bit and the corresponding pending bit is set, the function must send the MSI request at that time.

You should obtain the necessary MSI information (such as the message address and data) from the configuration output interface (tl_cfg_*) to create the MWr TLP in the format shown below to be sent via the Avalon^® -ST interface.

The following figure shows the timings of msi_pnd_* signals in three scenarios. The first scenario shows the case when the MSI pending bits register is not used. The second scenario shows the case when only physical function 0 is enabled and the MSI pending bits register is used. The last scenario shows the case when four physical functions are enabled and the MSI pending bits register is used.

There are 32 possible MSI messages. The number of messages requested by a particular component does not necessarily correspond to the number of messages allocated. For example, in the following figure, the Endpoint requests eight MSIs but is only allocated two. In this case, you must design the Application Layer to use only two allocated messages.

The following table describes three example implementations. The first example allocates all 32 MSI messages. The second and third examples only allocate 4 interrupts.

MSI interrupts generated for Hot Plug, Power Management Events, and System Errors always use Traffic Class 0. MSI interrupts generated by the Application Layer can use any Traffic Class. For example, a DMA that generates an MSI at the end of a transmission can use the same traffic control as was used to transfer data.

The following figure illustrates a possible implementation of the Interrupt Handler Module with a per vector enable bit in the Application Layer. Alternatively, the Application Layer could implement a global interrupt enable instead of this per vector MSI.

The P-Tile IP for PCIe provides a Configuration Intercept Interface. User soft logic can monitor this interface to get MSI-X Enable and MSI-X function mask related information. User application logic needs to implement the MSI-X tables for all PFs and VFs at the memory space pointed to by the BARs as a part of your Application Layer.

For more details on the MSI-X related information that you can obtain from the Configuration Intercept Interface, refer to the MSI-X Registers section in the Registers chapter.

MSI-X is an optional feature that allows the user application to support large amount of vectors with independent message data and address for each vector.

When MSI-X is supported, you need to specify the size and the location (BARs and offsets) of the MSI-X table and PBA. MSI-X can support up to 2048 vectors per function versus 32 vectors per function for MSI.

A function is allowed to send MSI-X messages when MSI-X is enabled and the function is not masked. The application uses the Configuration Output Interface (address 0x0C bit[5:4]) or Configuration Intercept Interface to access this information.

When the application needs to generate an MSI-X, it will use the contents of the MSI-X Table (Address and Data) and generate a Memory Write through the Avalon^® -ST interface.

You can enable MSI-X interrupts by turning on the Enable MSI-X option under the PCI Express/PCI Capabilities tab in the parameter editor. If you turn on the Enable MSI-X option, you should implement the MSI-X table structures at the memory space pointed to by the BARs as a part of your Application Layer.

The MSI-X Capability Structure contains information about the MSI-X Table and PBA Structure. For example, it contains pointers to the bases of the MSI-X Table and PBA Structure, expressed as offsets from the addresses in the function's BARs. The Message Control register within the MSI-X Capability Structure also contains the MSI-X Enable bit, the Function Mask bit, and the size of the MSI-X Table. For a picture of the MSI-X Capability Structure, refer to Figure 86.

MSI-X interrupts are standard Memory Writes, therefore Memory Write ordering rules apply.

Example:

If the application needs to generate an MSI-X interrupt (vector 1), it will read the MSI-X Table information, generate a MWR TLP through the Avalon^® -ST interface and assert the corresponding PBA bits (bit[1]) in a similar fashion as for MSI generation.

The generated TLP will be sent to address 0x00000001_BBBB0000 and the data will be 0x00000002. When the MSI-X has been sent, the application can clear the associated PBA bits.

This is an optional interface in the Intel P-Tile Avalon^® -ST IP for PCI Express that allows the Application Layer to report errors to the IP core and vice versa. Specifically, the Application Layer can report the different types of errors defined by the app_error_info_i signal to the IP. For Advanced Error Reporting (AER), the Application Layer can provide the information to log the TLP header and the error log request via the app_err_* interface.

The P-Tile IP for PCIe features a Completion timeout mechanism to keep track of Non-Posted requests sent by the user application and the corresponding Completions received. When the P-Tile IP detects a Completion timeout, it notifies the user application by asserting the cpl_timeout_o signal.

When a Completion timeout happens, the user application can use the Avalon^® -MM Completion Timeout Interface (for each port) to access the Completion timeout FIFO in the Hard IP to get more detailed information about the event and update the AER capability registers if required. After the completion timeout FIFO becomes empty, the IP core deasserts the cpl_timeout_o signal.

The cpl_timeout_avmm interface is synchronized to the cpl_timeout_avmm_clk_i clock.

Example:

When cpl_timeout_o is asserted, the user application can issue an Avalon^® -MM Read to retrieve information from the Completion FIFO. Then, it can issue an Avalon^® -MM Write to write 1 to bit[0] of the CONTROL register to get access to the next data.

Hot Plug support means that the device can be added to or removed from a system during runtime. The Hot Plug Interface in the P-Tile IP for PCIe allows an Intel FPGA with this IP to safely provide this capability.

This section describes the signals reported by the on-board hot plug components in the Downstream Port. This interface is available only if the Slot Status Register of the PCI Express Capability Structure is enabled.

Software programs the device into a D-state by writing to the Power Management Control and Status register in the PCI Power Management Capability Structure. The power management output signals indicate the current power state. The IP core supports the two mandatory power states: D0 (full power) and D3 (preparation for a loss of power). It does not support the optional D1 and D2 low-power states.

The correspondence between the device power states (D states) and link power states (L states) is as follows:

P-Tile does not support ASPM.

The Transaction Layer configuration output (tl_cfg) bus provides a subset of the information stored in the Configuration Space. Use this information in conjunction with the app_err* signals to understand TLP transmission problems.

The table below provides the tl_cfg_add_o[4:0] to tl_cfg_ctl_o[15:0] mapping.

The Configuration Intercept Interface (CII) allows the application logic to detect the occurrence of a Configuration (CFG) request on the link and to modify its behavior.

The application logic should detect the CFG request at the rising edge of cii_req. Due to the latency of the EMIB, the cii_req can be deasserted many cycles after the deassertion of cii_halt.

This interface also allows you to implement the Intel Vendor Specific Extended Capability (VSEC) registers. All configuration accesses targeting the Intel VSEC registers (addresses 0xD00 to 0xFFF) are automatically mapped to this interface and can be monitored via this interface.

If you are not using this interface, tie cii_halt_p0/1 to logic 0.

The Hard IP reconfiguration interface is an Avalon^® -MM slave interface with a 21‑bit address and an 8‑bit data bus. It is also sometimes referred to as the User Avalon^® -MM Interface. You can use this interface to dynamically modify the value of configuration registers. Note that after a warm reset or cold reset, changes made to the configuration registers of the Hard IP via the Hard IP reconfiguration interface are lost as these registers revert back to their default values.

In Root Port mode, the application logic uses the Hard IP reconfiguration interface to access its PCIe configuration space to perform link control functions (such as Hot Reset, link disable, or link retrain).

In TLP Bypass mode, the Hard IP forwards the received Type0/1 Configuration request TLPs to the application logic, which must respond with Completion TLPs with a status of Successful Completion (SC), Unsupported Request (UR), Configuration Request Retry Status (CRS), or Completer Abort (CA). If a received Configuration request TLP needs to update a PCIe configuration space register, the application logic needs to use the Hard IP reconfiguration interface to access that PCIe configuration space register.

Reading and Writing to the Hard IP Reconfiguration Interface

Reading from the Hard IP reconfiguration interface of the P-Tile Avalon^® -ST IP for PCI Express retrieves the current value at a specific address. Writing to the reconfiguration interface changes the data value at a specific address. Intel recommends that you perform read-modify-writes when writing to a register, because two or more features may share the same reconfiguration address.

Modifying the PCIe configuration registers directly affects the behavior of the PCIe device.

The User Avalon^® -MM interface provides access to the configuration registers and the IP core registers. This interface includes an 8-bit data bus and a 21-bit address bus (which contains the byte addresses).

Targeting PF Configuration Space Registers

User application needs to specify the offsets of the targeted PF registers.

For example, if the application wants to read the MSI Capability Register of PF0, it will issue a Read with address 0x0050 to target the MSI Capability Structure of PF0.

Targeting VF Configuration Space Registers

User application needs to first specify the VF number of the targeted configuration register.

The application needs to program the User Avalon-MM Port Configuration Register at offset 0x104068 accordingly.

Targeting VSEC Registers

User application needs to program the VSEC field (0x104068 bit[0]) first. Then all accesses from the user Avalon^® -MM interface starting at offset 0xD00 will be translated to VSEC configuration space registers.

DEBUG_DBI_ADDR register is located at user Avalon^® -MM offsets 0x104204 to 0x104207 (corresponding to byte 0 to byte 3).

DEBUG_DBI_DATA register is located at user Avalon^® -MM offsets 0x104200 to 0x104203 (corresponding to byte 0 to byte 3).

The PHY reconfiguration interface is an optional Avalon^® -MM slave interface with a 26‑bit address and an 8‑bit data bus. Use this bus to read the value of PHY registers. Refer to Table 101 for details on addresses and bit mappings for the PHY registers that you can access using this interface.

These signals are present when you turn on Enable PHY reconfiguration on the Top-Level Settings tab using the parameter editor.

Please note that the PHY reconfiguration interface is shared among all the PMA quads.

Reading from the PHY Reconfiguration Interface

Reading from the PHY reconfiguration interface of the P-Tile Avalon^® -ST IP for PCI Express retrieves the current value at a specific address.

When an Endpoint determines that it requires access to a page for which the ATS translation is not available, it sends a Page Request message to request that the page be mapped into system memory.

The PRS interface allows the monitoring of when PRS events happen, what functions these PRS events belong to, and what types of events they are.

The PRS interface is only available in EP mode, and with TLP Bypass disabled.

The figure below shows the timing diagram for the PRS event interface when the application layer of function 0 sends an event of PRG response reception, and the application layer of function 1 sends an event stopping requests for additional pages.

The PCIe* controller IP contains a set of port bifurcation muxes to remap the four controller PIPE lane interfaces to the shared 16 PCIe* PHY lanes. The table below shows the relationship between PHY lanes and the port mapping.

The P-Tile IP for PCIe supports SR-IOV. The endpoint port controllers in the IP support up to eight physical functions (PF) and 2048 virtual functions (VF) per SR-IOV endpoint. The VF configuration space registers are hardened in the P-Tile. The specific VF-based work queues and interrupt tables must be implemented in the FPGA fabric by the user application.

For more details on the configuration space registers required for virtualization support, refer to Configuration Space Registers for Virtualization.

The VF configuration space is implemented in P-Tile logic, and does not require FPGA fabric resources.

Accessing VF PCIe Information:

Due to the limited number of pins between P-Tile and the FPGA fabric, the PCIe configuration space for VFs is not directly available to the user application.

Accessing VF PCIe Information:

VF IDs are calculated within P-Tile. User application has sideband signals rx_st_vf_num_o and rx_st_vf_active_o with the TLP to identify the associated VFs within the PFs.

BDF Assignments:

When SR-IOV is enabled, the ARI capability is always enabled.

The P-Tile IP for PCIe automatically calculates the completer/requester ID on the Transmit side.

User application needs to provide the VF and PF information in the Header as shown below:

(For X16, sn is either s0 or s1. For X8, sn is s0).

• tx_st_hdr_sn[127]: must be set to 0
• tx_st_hdr_sn[83]: tx_st_vf_active
• tx_st_hdr_sn[82:80]: tx_st_func_num[2:0]
• tx_st_hdr_sn[95:84]: tx_st_vf_num[11:0]

In the following example, VF3 of PF1 is receiving and sending a request:

For the Receive TLP:

rx_st_func_num_o = 1h indicating that a VF associated with PF1 is making the request.

rx_st_vf_num_o = 3h, and rx_st_vf_active_o = 1 indicating that VF3 of PF1 is the active VF.

The VFs, with no AER support, are required to generate Non-Fatal error messages. The IP does not generate any error message. It is up to the user application logic to generate appropriate messages when specific error conditions occur.

VF to PF mapping always starts from the lowest possible PF number. For instance, if the IP has 2 PFs, wherein PF0 has 64 VFs and PF1 has 16 VFs, VF1 to VF64 are mapped to PF0, and VF65 to VF80 are mapped to PF1.

For example, the row that shows the combination of four PFs, 256 VFs, and the notation 256/0/0/0 in the Number of VFs per PF column indicates that all 256 VFs are mapped to PF0, while no VF is mapped to PF1, PF2 or PF3.

Use the FLR interface to reset individual SR-IOV functions. The PCIe* Hard IP supports FLR for both PFs and VFs. If the FLR is for a specific VF, the received packets for that VF are no longer valid. The flr_* interface signals are provided to the application interface for this purpose. When the flr_rcvd* signal is asserted, it indicates that an FLR is received for a particular PF/VF. Application logic needs to perform its FLR routine and send the completion status back on the flr_completed* interface. The Hard IP will wait for the flr_completed* status to re-enable the VF. Prior to that event, the Hard IP will respond to all transactions to the function that is reset by the FLR with completions with an Unsupported Request (UR) status.

The following figure shows the timing diagram for an FLR event targeting a PF (PF2 in this example):

Here is the timing diagram for an FLR event targeting a VF:

The VirtIO PCI configuration access capability creates an alternative access method to the common configuration, notifications, ISR, and device-specific configuration structure regions. This interface provides a means for the driver to access the VirtIO device region of Physical Functions (PFs) or Virtual Functions (VFs).

VirtIO is an industry standard for software-based virtualization that is supported natively by Linux. In VirtIO, software implements the virtualization stack, whereas in the case of SR-IOV, this stack is implemented mostly in hardware.

Below is the block diagram of the Soft IP which implements the VirtIO capability for PFs and VFs. This Soft IP block is automatically included when the VirtIO feature is enabled in the IP Parameter Editor.

For a detailed discussion of the VirtIO-related parameters, refer to the section VirtIO Parameters in the Parameters chapter.

To access a VirtIO device region, pci_cfg_data will provide a window of size cap.length (1, 2 or 4 Bytes) into the given cap.bar (0x0 – 0x5) at offset cap.offset (multiple of cap.length). Detailed interfaces mapping for the user application logic are shown in the following table.

As for the VirtIO device, upon detecting a driver write access to pci_cfg_data, the user application side's VirtIO device must execute a write access at cap.offset at the BAR selected by cap.bar using the first cap.length bytes from pci_cfg_data. Moreover, upon detecting a driver read access to pci_cfg_data, the user application side's VirtIO device must execute a read access of length cap.length at cap.offset at the BAR selected by cap.bar and store the first cap.length bytes in pci_cfg_data.

The following VirtIO capability structure registers references apply to each PF and VF. Addresses shown are register addresses.

The capability register identifies that this is a vendor-specific capability. It also identifies the structure type.

The Bar Indicator field holds the values 0x0 to 0x5 specifying a Base Address register (BAR) belonging to the function located beginning at 10h in PCI Configuration Space. The BAR is used to map the structure into the memory space. Any other value is reserved for future use.

This register indicates where the structure begins relative to the base address associated with the BAR. The alignment requirements of the offset are indicated in each structure-specific section.

The length register indicates the length of the structure. The length may include padding, fields unused by the driver, or future extensions.

The capability register identifies that this is a vendor-specific capability. It also identifies the structure type.

The Bar Indicator field holds the values 0x0 to 0x5 specifying a Base Address register (BAR) belonging to the function located beginning at 10h in PCI Configuration Space. The BAR is used to map the structure into memory space. Any other value is reserved for future use.

This register indicates where the structure begins relative to the base address associated with the BAR. The alignment requirements of the offset are indicated in each structure-specific section.

The length register indicates the length of the structure. The length may include padding, fields unused by the driver, or future extensions.

The notify off multiplier register indicates the multiplier for queue_notify_off in the structure.

The capability register identifies that this is a vendor-specific capability. It also identifies the structure type.

The Bar Indicator field holds the values 0x0 to 0x5 specifying a Base Address register (BAR) belonging to the function located beginning at 10h in PCI Configuration Space. The BAR is used to map the structure into memory space. Any other value is reserved for future use.

This register indicates where the structure begins relative to the base address associated with the BAR. The alignment requirements of the offset are indicated in each structure-specific section.

The length register indicates the length of the structure. The length may include padding, fields unused by the driver, or future extensions.

The capability register identifies that this is vendor-specific capability. It also identifies the structure type.

The BAR Indicator field holds the values 0x0 to 0x5 specifying a Base Address register (BAR) belonging to the function located beginning at 10h in PCI Configuration Space. The BAR is used to map the structure into memory space. Any other value is reserved for future use.

This register indicates where the structure begins relative to the base address associated with the BAR. The alignment requirements of the offset are indicated in each structure-specific section.

The length register indicates the length of the structure. The length may include padding, fields unused by the driver, or future extensions.

The capability register identifies that this is a vendor-specific capability. It also identifies the structure type.

The BAR Indicator field holds the values 0x0 to 0x5 specifying a Base Address register (BAR) belonging to the function located beginning at 10h in PCI Configuration Space. The BAR is used to map the structure into memory space. Any other value is reserved for future use.

This register indicates where the structure begins relative to the base address associated with the BAR. The alignment requirements of the offset are indicated in each structure-specific section.

The length register indicates the length of the structure. The length may include padding, fields unused by the driver, or future extensions.

The PCI configuration data register indicates the data for BAR access.

When the TLP Bypass feature is enabled, the P-Tile Avalon^® -ST IP does not process received TLPs internally but outputs them to the user application. This allows the application to implement a custom Transaction Layer.

Note that in TLP Bypass mode, the PCIe Hard IP does not generate/check the ECRC and will not remove it if the received TLP has the ECRC.

The P-tile Avalon^® -ST IP in TLP Bypass mode still includes some of the PCIe configuration space registers related to link operation (refer to the Configuration Space Registers chapter for the list of registers).

In TLP bypass mode, P-Tile supports the autonomous Hard IP feature. It responds to configuration accesses before the FPGA fabric enters user mode with Completions with a CRS code.

However, in TLP bypass mode, CvP init and update are not supported.

When TLP Bypass mode is enabled, some error detection is still performed in the Physical and Link Layers inside the Hard IP. Per PCIe specification, the Hard IP must report these errors on the configuration space registers (in the AER Capability Structure). The P-tile IP for PCIe includes two registers called TLPBYPASS_ERR_EN and TLPBYPASS_ERR_STATUS to report errors detected while in TLP Bypass mode.

TLPBYPASS_ERR_EN and TLPBYPASS_ERR_STATUS are part of the configuration and status register.

This register allows you to enable or disable error reporting. When this feature is disabled, the TLPBYPASS_ERR_ STATUS bits associated with an error are not set when the error is detected.

When an error is detected, Intel recommends that you read the PF0 AER register inside P-Tile to get detailed information about the error. To clear the previous error status, you need to clear TLPBYPASS_ERR_STATUS and the corresponding correctable and uncorrectable error status registers in the AER capability structure. After doing that, you can get the new error update from this register.

For more details on the signals in this interface, refer to the section Hard IP Reconfiguration Interface.

The majority of the PCIe standard registers are implemented in the user’s logic outside of the P-Tile Avalon^® -ST IP.

The application can only access PCIe controller registers through the User Avalon^® -MM interface.

For more details on the signals in this interface, refer to the section Avalon-ST Interface.

The P-Tile IP forwards any received Type0/1 Configuration TLP to the Avalon^® -ST RX streaming interface. User’s logic has the responsibility to respond with a Completion TLP with a Completion code of Successful Completion (SC), Unsupported Request (UR), Configuration Request Retry Status (CRS), or Completer Abort (CA).

If a Configuration TLP needs to update a register in the PCIe configuration space in the P-Tile PCIe Hard IP, you need to use the User Avalon^® -MM interface.

The application needs to prevent link programming side effects such as writing into low-power states before sending the Completion associated with the request. The application logic can check the TX FIFO empty flag in the tx_cdts_limit_o after the Completion enters the TX streaming interface to confirm that the TLP has been sent. For more details on the User Avalon^® -MM interface, refer to the section Hard IP Reconfiguration Interface (User Avalon^® -MM Interface).

ALL TLPs received by the IP are transmitted to the application through the RX streaming interface (except Malformed TLPs).

Please refer to the Packets Forwarded to the User Application in TLP Bypass Mode Appendix for detailed information.

All PCIe protocol errors leading up to designating a TLP packet as a good packet or not will be detected by the Hard IP and communicated to user logic to take appropriate action in terms of error logging and escalation. The IP does not generate any error message internally, since this is the responsibility of the user logic.

In TLP Bypass mode, a malformed TLP is dropped in the P-Tile IP for PCIe and its event is logged in the AER capability registers. P-Tile also notifies you of this event by asserting the serr_out_o signal.

Refer to the PCI Express Base Specification for the definition of a malformed TLP.

This chapter introduces the testbench for an Endpoint design example and a test driver module. You can create this design example using design flows described in Quick Start Guide chapter of the Intel FPGA P-Tile Avalon streaming IP for PCI Express Design Example User Guide.

The testbench in this design example simulates up to a Gen4 x16 variant.

When configured as an Endpoint variation, the testbench instantiates a design example with a P-Tile Endpoint and a Root Port BFM containing a second P-Tile (configured as a Root Port) to interface with the Endpoint. The Root Port BFM provides the following functions:

• A configuration routine that sets up all the basic configuration registers in the Endpoint. This configuration allows the Endpoint application to be the target and initiator of PCI Express transactions.
• A Verilog HDL procedure interface to initiate PCI Express* transactions to the Endpoint.

This testbench simulates the scenario of a single Root Port talking to a single Endpoint.

The testbench uses a test driver module, altpcietb_bfm_rp_gen4_x16.sv, to initiate the configuration and memory transactions. At startup, the test driver module displays information from the Root Port and Endpoint Configuration Space registers, so that you can correlate to the parameters you specified using the Parameter Editor.

Your Application Layer design may need to handle at least the following scenarios that are not possible to create with the Intel testbench and the Root Port BFM, or are due to the limitations of the example design:

• It is unable to generate or receive Vendor Defined Messages. Some systems generate Vendor Defined Messages. The Hard IP block simply passes these messages on to the Application Layer. Consequently, you should make the decision, based on your application, whether to design the Application Layer to process them.
• It can only handle received read requests that are less than or equal to the currently set Maximum payload size option specified under the Device tab under the PCI Express/PCI Capabilities GUI using the parameter editor. Many systems are capable of handling larger read requests that are then returned in multiple completions.
• It always returns a single completion for every read request. Some systems split completions on every 64-byte address boundary.
• It always returns completions in the same order the read requests were issued. Some systems generate the completions out-of-order.
• It is unable to generate zero-length read requests that some systems generate as flush requests following some write transactions. The Application Layer must be capable of generating the completions to the zero-length read requests.
• It uses fixed credit allocation.
• It does not support parity.
• It does not support multi-function designs.
• It incorrectly responds to Type 1 vendor-defined messages with CplD packets.

The example design and testbench are dynamically generated based on the configuration that you choose for the P-Tile IP for PCIe. The testbench uses the parameters that you specify in the Parameter Editor in Intel^® Quartus^® Prime.

This testbench simulates up to a ×16 PCI Express link using the serial PCI Express interface. The testbench design does allow more than one PCI Express link to be simulated at a time. The following figure presents a high level view of the design example.

The top-level of the testbench instantiates the following main modules:

• altpcietb_bfm_rp_gen4_x16.sv —This is the Root Port PCIe* BFM.

  //Directory path
  <project_dir>/intel_pcie_ptile_ast_0_example_design/pcie_ed_tb/ip/pcie_ed_tb/dut_pcie_tb_ip/intel_pcie_ptile_tbed_<ver>/sim

• pcie_ed_dut.ip: This is the Endpoint design with the parameters that you specify.

  //Directory path
  <project_dir>/intel_pcie_ptile_ast_0_example_design/ip/pcie_ed

• pcie_ed_pio0.ip: This module is a target and initiator of transactions for the PIO design example.

  //Directory path
  <project_dir>/intel_pcie_ptile_ast_0_example_design/ip/pcie_ed

• pcie_ed_sriov0.ip: This module is a target and initiator of transactions for the SR-IOV design example.

  //Directory path
  <project_dir>/intel_pcie_ptile_ast_0_example_design/ip/pcie_ed

In addition, the testbench has routines that perform the following tasks:

• Generates the reference clock for the Endpoint at the required frequency.
• Provides a PCI Express reset at start up.

The SR-IOV design example testbench supports up to two Physical Functions (PFs) and 32 Virtual Functions (VFs) per PF.

For more details on the PIO design example testbench and SR-IOV design example testbench, refer to the Intel FPGA P-Tile Avalon^® streaming IP for PCI Express Design Example User Guide.