Intel® Acceleration Stack for Intel® Xeon® CPU with FPGAs 1.2.1 and 2.0.1
1. About this Document
This document describes how to simulate a sample Accelerator Functional Unit (AFU) using the
Intel® Accelerator Functional Unit (AFU) Simulation Environment (ASE) environment.
Refer to the
Intel® Accelerator Functional Unit (AFU) Simulation Environment (ASE) User
Guide for comprehensive details on ASE
capabilities and internal architecture.
Intel® Accelerator Functional Unit (AFU) Simulation Environment (ASE) is a hardware and
software co-simulation environment for any
FPGA Programmable Acceleration Card (Intel® FPGA PAC).
This software co-simulation environment currently supports the following
Intel® FPGA PACs:
Intel® Programmable Acceleration Card with Intel®
Arria® 10 GX FPGA
Intel FPGA Programmable Acceleration Card D5005
The ASE provides a transactional model
for the Core Cache Interface (CCI-P) protocol and a memory model for the
FPGA-attached local memory.
The ASE also validates Accelerator Functional Unit (AFU) compliance to the following protocols and APIs:
The CCI-P protocol
Avalon® Memory Mapped (
Avalon®-MM) Interface Specification
The Open Programmable Acceleration Engine (OPAE)
Table 1. Acceleration Stack for
with FPGAs Glossary
Intel® Acceleration Stack for Intel®
Xeon® CPU with FPGAs
A collection of software, firmware and tools that provides
performance-optimized connectivity between an
Intel® FPGA and an
Quartus® Prime Pro Edition
1 The required version is installed with the Acceleration Stack for
3. Setting Up the Environment
You must set up your simulation
environment and install the OPAE software before
running the ASE.
Set the following environment variables for your simulation
$ export VCS_HOME=<path to VCS installation directory>
$ export PATH=$VCS_HOME/bin:$PATH
The VCS installation directory structure is as
Make sure your system has a valid VCS
$ export MTI_HOME=<path to Modelsim installation directory>
$ export PATH=$MTI_HOME/linux_x86_64/:$MTI_HOME/bin/:$PATH
The Modelsim/Questa installation directory
structure is as follows:
Make sure your system has a valid Modelsim
Quartus® Prime Pro Edition:
$ export QUARTUS_HOME=<path to Intel Quartus Prime Pro Edition installation directory>
installation directory structure is as follows:
Add the environment variable to check the Modelsim
$ export MGLS_LICENSE_FILE=<Modelsim License>
$ export LM_LICENSE_FILE=<Quartus Prime License>
Extract the runtime archive file, and install OPAE libraries, binaries, include files, and
ASE libraries as described in the
section: Installing the OPAE Software Package in the appropriate Intel
Acceleration Stack Quick Start User Guide for your
Intel® FPGA PAC.
Your environment must be set up correctly to configure and build an
AFU. In particular, you must install the
OPAE Software Development Kit (SDK)
properly. OPAE SDK scripts must be on PATH and include files and libraries that must be
available to the C compiler. In addition, you must ensure that the OPAE_PLATFORM_ROOT environment variable is set. Refer
to Installing the OPAE
Software Package for more information.
To ensure that the OPAE SDK and
ASE are properly installed, in a shell,
confirm that your PATH includes afu_sim_setup. The afu_sim_setup
should be in the /usr/bin directory or in <opae install
path> if you built the OPAE from source files.
The hello_afu example is a simple AFU template that
demonstrates the primary CCI-P interface. The RTL satisfies the minimum requirements of an
AFU, responding to memory-mapped I/O reads to return the device feature header and the AFU's
Figure 1. hello_afu Directory Tree
Note: This document uses <AFU example> to refer to an example design directory, such
as hello_afu in the figure above.
The software demonstrates the minimum requirements to attach to an FPGA
the OPAE. The RTL demonstrates the minimum requirements to satisfy the OPAE
driver and the hello_afu example software.
filelist.txt specifies the files for
RTL simulation and synthesis.
To successfully configure and build the AFU samples, your environment must be
set up correctly, as described in Setting Up the
The following example flow introduces the basic ASE scripts. You can simulate all examples with the ASE, except eth_e2e_e10
Simulation requires two software processes: one process for RTL simulation and
a second process to run the connected software. To construct an RTL simulation environment,
run the following in $OPAE_PLATFORM_ROOT/hw/samples/hello_afu:
example> is the example directory as shown in the hello_afu Directory Tree figure.
filelist.txt lists SystemVerilog,
The AFU.json describes the interfaces the AFU requires. It also includes a UUID to identify the AFU once downloaded to an FPGA.
hw/rtl/hello_afu.json defines ccip_std_afu as the top-level interface by setting afu-top-interface to ccip_std_afu. ccip_std_afu is the base CCI-P interface including clocks, reset, and CCI-P TX and RX structures. More advanced examples define
other interface options.
The .json file declares the AFU
UUID. An OPAE script generates the UUID. The RTL loads
the UUID from afu_json_info.vh.
sw/Makefile generates afu_json_info.h. Software loads the UUID from afu_json_info.h.
4.1.3. Troubleshooting Client-Server Simulation
If the afu_sim_setup command fails, confirm
afu_sim_setup is on your PATH. afu_sim_setup should be
in /usr/bin or in <opae
install path> if you built OPAE from
You have Python version 2.7 or higher
If you are unable to build and execute the simulator, it is likely that you
did not install your RTL simulation tool properly.
When you try to build and run the software, if you see an "Error enumerating
AFCs" message, you omitted setting USE_ASE=1 on the make command line. The software is
searching for a physical FPGA device. To recover, repeat the steps from the make clean command.
5. AFU Examples
Table 2. AFU ExamplesEach AFU example includes a detailed README file, providing
an operational description and notes on how to simulate the design. For a full
understanding of the simulation process, review the README file in each AFU
hello_mem_afu demonstrates an AFU that
builds a simple state machine to access memory. The state machine is
capable of several access patterns to local memory directly attached
to FPGA pins, such as DDR4 DIMMs. This memory is distinct from the
host memory accessed over CCI-P. The host manages the hello_mem_afu controller state machine
using memory-mapped I/O (MMIO) requests to control and status
hello_intr_afu demonstrates the
application interrupt feature in the ASE.
dma_afu demonstrates a DMA Basic
Building Block for host to FPGA, FPGA to host, and FPGA to FPGA
memory transfers. When simulating this AFU, the buffer size used for DMA transfer is small
to keep the simulation time reasonable. For more information, refer
to the DMA Accelerator Functional Unit (AFU) User Guide.
nlb_mode_0 is a CCI-P system demonstrating the
memory copy test. $OPAE_PLATFORM_ROOT/sw/opae-<release_number>/sample/hello_fpga.c
$ sh regress.sh -a <afu dir> -r rtl_sim
-s < vcs|modelsim|questa > [-i <opae install path>]
-b <path to opae source dir>
demonstrates how to transfer data between host memory and an
FPGA streaming port. For more information, refer to the Streaming DMA Accelerator Functional Unit (AFU) User Guide.
hello_afu is a simple AFU that demonstrates the primary
CCI-P interface. The RTL satisfies the bare minimum requirements of
an AFU, responding to MMIO
reads to return the device feature header and the AFU's UUID.
If the following error appears during simulation, correct it by
following the steps below.
# [SIM] An ASE instance is probably still running in current directory !
# [SIM] Check for PID 28816
# [SIM] Simulation will exit... you may use a SIGKILL to kill the simulation process.
# [SIM] Also check if .ase_ready.pid file is removed before proceeding.
Type pkill ase_simv to kill zombie
simulation processes and remove any temporary files left behind by failed
simulation processes or lock ups.
Delete the .ase_ready.pid file, found in