The F-Tile Avalon-ST IP for PCI Express Design Example is a simple design to
demonstrate the establishment of PCIe connectivity of F-Tile FPGA in
Quartus® Prime. The design performs write and read sequences from the host processor
to the target device through PCIe
Quartus® Prime Hard IP. The
Programmed Input/Output (PIO) application block is needed to handle the translation from
PCIe TLP to AVMM protocol.
Table 2. Configurations Supported by the F-Tile Avalon-ST Design
Note: Gen1/Gen2 x1/x2
configurations are supported via link down-training.
Note: N/A =
Note: In the 21.2 release of
Quartus® Prime, this design example only supports the
default settings in the Parameter Editor of the F-tile Avalon Streaming IP for PCI
2.1. Functional Description for the Programmed Input/Output Design Example
Figure 1. PCIe Gen3/Gen4
x16 Design Example Variant Block
Figure 2. PCIe Gen3/Gen4
Design Example Variant
The PCIe F-Tile Design Example is designed to highlight the application of
PCIe Gen4 in F-Tile.
The PCIe IP run up to
500MHz at user interface with the maximum data width of 512 bits for Gen4 x16 and 256 bits
for each of the two Gen4 x8
The clock frequency supplied from the coreclkout_hip is limited to 500 MHz.
The Design Example consists of 3 main components to display the intended
IP for PCI Express Hard
Programmable I/O Application (PIO)
On-Chip Memory (MEM)
The PIO design example performs memory transfers from a host processor to a
target device. In this example, the host processor requests single-dword MemRd and MemWr
The PIO design example automatically creates the
simulate and compile in the
Quartus® Prime software. The design
example covers a wide range of parameters. However, it does not cover all possible
parameterizations of the F-Tile Hard IP for PCIe.
This design example includes the following components:
Avalon-ST IP for PCI Express Hard
(DUT) with the parameters you specified. This component drives TLP data received to the
PIO application. The DUT component is the
IP for PCI Express Hard IP
configured as Endpoint interacting with the root complex/switch at the other end. The
the PCIe serial link transfer interface to
The PIO Application (APPS) component performs the necessary
translation between the PCI Express TLPs and simple Avalon-MM writes and reads to the
on-chip memory. The PIO component interfaces between the
It decodes the TLP headers/data and converts it into
An on-chip memory (MEM) component stores and reads data depending on
the instructions from
PIO. The size of
the on-chip memory is configured to 16 KB for both PCIe Gen3/4 x16 and PCIe Gen3/4 x8x8
design example variants.
and System PLL Clocks IP: This IP is required for F-Tile PCIe interface implementation
to configure the reference clock for the FGT PMA and System PLL. The clock from this IP
is an logical connection. It is physically inside the F-Tile Avalon-ST IP for PCI
Express Hard IP. There is no clock gating requirement at the design example level. The
main clock of the PIO design example originates from coreclkout_hip of F-Tile Avalon-ST
IP for PCI Express Hard IP running at 500 MHz. The clock is originates from System PLL.
This IP is required for F-Tile PCIe interface implementation to configure the reference
clock for the FGT PMA amd System PLL.
Reset Release IP: This IP holds a control circuit in reset until the
device has fully entered user mode. The FPGA asserts the INIT_DONE output to signal that the device is in user mode. The Reset
Release IP generates an inverted version of the internal INIT_DONE signal to create the nINIT_DONE
output that you can use for your design.The nINIT_DONE
signal is high until the entire device enters user mode. After nINIT_DONE asserts (low), all logic is in user mode and operates normally.
You can use the nINIT_DONE signal in one of the
To gate an external or internal reset.
To gate the reset input to I/O PLLs.
To gate the write enable of design blocks such as embedded memory
blocks, state machine, and shift registers.
To synchronously drive register reset input ports in your
Note: For more information on
Reset Release IP, refer to Intel Agilex Configuration User
Note: For more information
about the F-Tile Reference and System PLL Clocks IP, refer to F-Tile Architecture and PMA and FEC Direct PHY IP User Guide
The simulation testbench instantiates the PIO design example and a Root
Port BFM to interface with the target Endpoint.
Figure 3. Block Diagram for the PCIe x16 PIO Design Example Simulation
Figure 4. Block Diagram for the PCIe x8x8 PIO Design Example Simulation
Note: The simulation testbench for PCIe x8x8 PIO Design Example is configured for a single
PCIe x8 link although the actual design implements two PCIe x8 links.
The test program writes to and reads back data from the same location in
the on-chip memory. It compares the data read to the expected result. The test reports,
"Simulation stopped due to successful
completion" if no errors occur.
Note: In the
Quartus® Prime, this design example only supports
the default settings in the Parameter Editor of the F-tile Avalon Streaming IP for PCI
Figure 5. Platform Designer System Contents for
Avalon®-ST IP for PCI Express PIO Design
Example The Platform Designer generates this
design for up to Gen4 x16 variants.
Figure 6. Platform Designer System Contents for
Avalon®-ST IP for PCI Express PIO Design
The Platform Designer generates this
design for up to Gen4 x8x8 variants.
Quartus® Prime software, you can
generate a programmed I/O (PIO) design example for the
Intel® FPGA F-Tile
IP core. The generated design
example reflects the parameters that you specify. The PIO example transfers data from a host
processor to a target device. It is appropriate for low-bandwidth applications. This design
example automatically creates the files necessary to simulate and compile in the
Quartus® Prime software. You can download the compiled design to
your FPGA Development Board. To download to custom hardware, update the
Quartus® Prime Settings File (.qsf) with the correct pin assignments .
Figure 7. Development Steps for the Design Example
3.1. Directory Structure
Figure 8. Directory Structure for the Generated Design Example
3.2. Generating the Design Example
Figure 9. Procedure
Quartus® Prime Pro Edition
software, create a new project (File > New Project Wizard).
Specify the Directory,
Name, and Top-Level
For Project Type,
accept the default value, Empty project.
For Add Files click
For Family, Device & Board
Settings under Family,
Select the Target
Device for your design.
In the IP Catalog locate and add the Intel F-Tile
Avalon®-STHard IP for
In the New IP Variant
dialog box, specify a name for your IP. Click Create.
On the Top-Level
tabs, specify the parameters for your IP variation.
Note: In the
this design example only supports the default settings in the Parameter
Editor of the F-tile Avalon® Streaming IP for PCIe.
On the Example Designs
tab, make the following selections:
For Example Design
Files, turn on the Simulation and Synthesis options. If you do not need these simulation
or synthesis files, leaving the corresponding option(s) turned off
significantly reduces the example design generation time.
For Generated HDL
Format, only Verilog is available in the current
For Target Development
F-Series P-Tile ES0 FPGA Development Kit.
Select Generate Example
Design to create a design example that you can simulate and
download to hardware. If you select one of the
development boards, the device on that board overwrites the device previously
selected in the
Quartus® Prime project if the
devices are different. When the prompt asks you to specify the directory for
your example design, you can accept the default directory, ./pcie_avst_f_0_example_design,
or choose another directory.
Figure 10. Example Designs Tab
Click Finish. You may
save your .ip file when prompted, but it
is not required to be able to use the example design.
3.3. Simulating the Design Example
Figure 11. Procedure
directory <example_design> and run the
command sh ip_sim.sh to generate the <example_design>/sim directory.
Run the simulation script
<example_design>/sim directory for the simulator of your choice.
Refer to the table below.
Analyze the results.
Note: F-Tile does not support
parallel PIPE simulations.
Table 3. Steps to Run Simulation
sh vcs_setup.sh USER_DEFINED_COMPILE_OPTIONS="" USER_DEFINED_ELAB_OPTIONS="+vcs+lic+wait\ -full64\ -hsopt=gates\ -debug_pp\ +define+RTLSIM\ +define+SSM_SEQUENCE\ " USER_DEFINED_SIM_OPTIONS="" | tee simulation.log
Note: The command above is a single-line command.
A successful simulation ends with the following message,
"Simulation stopped due to successful completion!"
the simulation.log file that was generated.
Note: To run a simulation in interactive mode, use the following steps: (if you
already generated a simv executable in noninteractive mode, delete the simv
Open the vcs_setup.sh file and add a debug option to the VCS
Compile the design example:
sh vcs_setup.sh USER_DEFINED_ELAB_OPTIONS="+vcs+lic+wait\ -full64\ -hsopt=gates\ -debug_pp\ +define+RTLSIM\ +define+SSM_SEQUENCE\ " SKIP_SIM=1
Start the simulation in interactive mode:
simv -gui &
Invoke vsim (by typing vsim, which brings up a console
window where you can run the following commands).
A successful simulation ends with the following message:
"Simulation stopped due to successful completion!"
The simulation reports, "Simulation stopped
due to successful completion" if no errors occur.
The same procedure applicable for both both PCIe Gen3/4 x16 and PCIe Gen3/4 x8x8 design
The testbench uses a test driver module, altpcietb_bfm_rp_gen4_x16.sv, to initiate the configuration and memory
transactions. At startup, the test driver module displays information from the Root
Port and Endpoint Configuration Space registers, so that you can correlate to the
parameters you specified using the Parameter Editor.
The example design and testbench are dynamically generated based on the
configuration that you choose for the F-Tile IP for PCIe. The testbench uses the
parameters that you specify in the Parameter Editor in
This testbench simulates up to a x16 PCI Express link using the
serial PCI Express interface. The testbench design does allow more than one PCI
Express link to be simulated at a time. The following figure presents a high level
view of the PIO design example.
Figure 12. PIO Design Example Simulation Testbench
The top-level of the testbench instantiates the following main
altpcietb_bfm_rp_gen4x16.sv —This is
the Root Port
In addition, the testbench has routines that perform the following
Generates the reference
clock for the Endpoint at the required frequency.
Provides a PCI Express
reset at start up.
For more details on the Root Port BFM, refer to the TestBench chapter of the Intel
Avalon® Streaming IP for PCI
Express User Guide.
188.8.131.52. Test Driver Module
The test driver module, intel_pcie_ftile_tbed_hwtcl.v,
instantiates the top-level BFM,altpcietb_bfm_top_rp.v.
The top-level BFM completes the following tasks:
Instantiates the driver and monitor.
Instantiates the Root Port BFM.
Instantiates the serial interface.
The configuration module, altpcietb_g3bfm_configure.v, performs the following tasks:
Configures and assigns the BARs.
Configures the Root Port and Endpoint.
Displays comprehensive Configuration Space, BAR, MSI, MSI-X,
and AER settings.
184.108.40.206.1. PIO Design Example Testbench
The figure below shows the PIO design example simulation design
hierarchy. The tests for the PIO design example are defined with the apps_type_hwtcl parameter set to 3. The tests run under
this parameter value are defined in ebfm_cfg_rp_ep_rootport, find_mem_bar
Figure 13. PIO Design Example Simulation Design Hierarchy
The testbench starts with link training and then accesses the
configuration space of the IP for enumeration. A task called downstream_loop (defined in the Root Port PCIe BFM altpcietb_bfm_rp_gen4_x16.sv) then performs the PCIe
link test. This test consists of the following steps:
Issue a memory write command to write a single dword of data
into the on-chip memory behind the Endpoint.
Issue a memory read command to read back data from the
Compare the read data with the write data. If they match,
the test counts this as a Pass.
Repeat Steps 1, 2 and 3 for 10 iterations.
The first memory write takes place around
µs. It is followed by a memory read at the
Avalon®-ST RX interface of the F-tile Hard IP for PCIe. The Completion TLP appears
shortly after the memory read request at the
Avalon®-ST TX interface. The memory write and read transactions and
the Completion TLP are shown in the following waveforms.
Figure 14. Simulation Waveforms for the PIO Design Example for the
Avalon®-ST IP for PCIe
3.4. Compiling the Design Example
Navigate to <project_dir>/pcie_avst_f_0_example_design/
and open pcie_ed.qpf.
On the Processing menu, select Start
Open the example design project.
Compile the example design project examine the design
compilation result like resource utilization and timing result.
Note: Hardware support or the .sof configuration file generation may be
supported in a future release.
Close your example design project.
cannot change the PCIe pin allocations in the
Quartus® Prime project. However, to ease PCB routing, you can take
advantage of the lane reversal and polarity inversion features supported by
3.5. Installing the Linux Kernel Driver
Before you can test the design example in hardware, you must install the Linux kernel
driver. You can use this driver to perform the following tests:
A PCIe link test that performs 100 writes and reads
Memory space DWORD(1) reads and writes
Configuration Space DWORD reads and writes
Note: Throughout this user guide, the term word DWORD has the same meaning as in the
PCI Express Base Specification. A word is 16 bits and a DWORD is 32 bits.
In addition, you can use the driver to change the value of the following parameters:
The BAR being used
The selected device (by specifying the bus, device and function (BDF) numbers for the
Complete the following steps to install the kernel driver
Navigate to ./software/kernel/linux under the example design
Change the permissions on the install, load, and unload
$ chmod 777 install load unload
Install the driver
$ sudo ./install
Verify the driver
$ lsmod | grep intel_fpga_pcie_drv
intel_fpga_pcie_drv 17792 0
Verify that Linux recognizes the PCIe design example
$ lspci -d 1172:000 -v | grep intel_fpga_pcie_drv
Expected result. Kernel driver in use:
Note: If you have changed the Vendor ID,
substitute the new Vendor ID for Intel's Vendor ID in this command.
3.6. Running the Design Example
Table 4. Test Operations Supported by the F-Tile Avalon-ST IP for PCI Express Design
Supported by F-Tile Avalon-ST IP for PCI Express Design Example
0: Link test - 100 writes and reads
1: Write memory space
2: Read memory space
3: Write configuration space
4: Read configuration space
5: Change BAR
6: Change device
7: Enable SR-IOV
8: Do a link test for every enabled virtual function belonging to the current
9: Perform DMA
10: Quit program
3.6.1. Running the PIO Design Example
Navigate to ./software/user/example under the design
Compile the design example application: $ make
Run the test: $ sudo ./intel_fpga_pcie_link_test
You can run the Intel FPGA IP PCIe link test in manual or automatic mode.
In automatic mode, the application automatically selects the
device. The test selects the Intel PCIe device with the lowest
BDF by matching the Vendor ID. The test also selects the lowest
In manual mode, the test queries you for the bus, device, and
function number and BAR.
For the Intel Agilex Development Kit, you can determine the BDF by
typing the following command:
$ lspci -d 1172
Here are sample transcripts for automatic and manual modes.
Figure 15. Automatic Mode
Figure 16. Manual
4. Revision History for the F-Tile Avalon® Streaming Intel® FPGA IP for PCI Express* Design Example User Guide