F-Tile Avalon® Streaming Intel® FPGA IP for PCI Express* User Guide

ID 683140
Date 8/19/2024
Public
Document Table of Contents

3.4.3. MSI-X

The F-Tile IP for PCIe provides a Configuration Intercept Interface. User soft logic can monitor this interface to get MSI-X Enable and MSI-X function mask related information. User application logic needs to implement the MSI-X tables for all PFs and VFs at the memory space pointed to by the BARs as a part of your Application Layer.

For more details on the MSI-X related information that you can obtain from the Configuration Intercept Interface, refer to the MSI-X Registers section in the Configuration Register section.

MSI-X is an optional feature that allows the user application to support large amount of vectors with independent message data and address for each vector. When MSI-X is supported, you need to specify the size and the location (BARs and offsets) of the MSI-X table and Pending Bit Array (PBA). MSI-X can support up to 2048 vectors per function versus 32 vectors per function for MSI. A function is allowed to send MSI-X messages when MSI-X is enabled and the function is not masked. The application uses the Configuration Output Interface (address 0x0C bit[5:4]) or Configuration Intercept Interface to access this information.

When the application needs to generate an MSI-X, it uses the contents of the MSI-X Table (Address and Data) and generate a Memory Write through the Avalon-ST interface.

If user enable MSI-X interrupt, you should implement the MSI-X table structures at the memory space pointed to by the BARs as a part of your Application Layer. The MSI-X Capability Structure contains information about the MSI-X Table and PBA Structure. For example, it contains pointers to the bases of the MSI-X Table and PBA Structure, expressed as offsets from the addresses in the function's BARs. The Message Control register within the MSI-X Capability Structure also contains the MSI-X

MSI-X interrupts are standard Memory Writes, therefore Memory Write ordering rules apply.

Table 17.  MSI-X Configuration Example
MSI-X Vector MSI-X Upper Address MSI-X Lower Address MSI-X Data
0 0x00000001 0xAAAA0000 0x00000001
1 0x00000001 0xBBBB0000 0x00000002
2 0x00000001 0xCCCC0000 0x00000003
Table 18.  PBA Table Example
PBA Table PBA Entries
Offset 0 0x0

If the application needs to generate an MSI-X interrupt (vector 1), it reads the MSI-X Table information, generates a MWR TLP through the Avalon-ST interface and asserts the corresponding PBA bits (bit[1]) in a similar fashion as for MSI generation.

The generated TLP is sent to address 0x00000001_BBBB0000 and the data is 0x00000002. When the MSI-X has been sent, the application can clear the associated PBA bits.

The MSI-X capability structure points to the MSI-X Table structure and MSI-X Pending Bit Array (PBA) registers. The BIOS sets up the starting address offsets and BAR associated with the pointer to the starting address of the MSI-X Table and PBA registers.

Figure 33. MSI-X Interrupt Component
  1. Host software sets up the MSI-X interrupts in the Application Layer by completing the following steps:
    • Host software reads the Message Control register at 0x050 register to determine the MSI-X Table size. The number of table entries is the <value read> + 1. The maximum table size is 2048 entries. Each 16-byte entry is divided in 4 fields as shown in the figure below. For multi-function variants, BAR4 accesses the MSI-X table. For all other variants, any BAR can access the MSI-X table. The base address of the MSI-X table must be aligned to a 4 KB boundary.
    • The host sets up the MSI-X table. It programs MSI-X address, data, and masks bits for each entry as shown in the figure below.
      Figure 34. Format of MSI-X Table
    • The host calculates the address of the <nth> entry using the following formula:
      nth_address = base address[BAR]+16<n>
  2. When Application Layer needs to issue an interrupt, it drives an interrupt request to the IRQ Source module.
  3. The IRQ Sources sets appropriate bit in the MSI-X PBA table. The PBA can use qword or dword accesses. For qword accesses, the IRQ Source calculates the address of the <mth> bit using the following formulas:
    qword address = <PBA base addr> + 8(floor(<m>/64))
    qword bit = <m> mod 64
    
    Figure 35. MSI-X PBA Table
  4. The IRQ Processor reads the entry in the MSI-X Table.
    • If the interrupt is masked by the Vector_Control field of the MSI-X table, the interrupt remains in the pending state.
    • If the interrupt is not masked, IRQ Processor sends Memory Write Request to the TX slave interface. It uses the address and data from the MSI-X table. If Message Upper Address =0, the IRQ Processor creates a three-dword header. If the Message Upper Address > 0, it creates a 4-dword header.
  5. The host interrupt service routine detects the TLP as an interrupt and service it.