Hierarchical Partial Reconfiguration over PCI Express Reference Design for Arria 10 Devices
The Hierarchical Partial Reconfiguration (HPR) over
PCI Express® (
design demonstrates reconfiguring the FPGA fabric through the
PCIe® link in
Arria® 10 devices.
This reference design runs on a Linux system with the
GX FPGA development board. Adapt this reference design to your requirements by implementing the
PR region logic using the given template. Run your custom design on this fully functional system
that enables communication over
Arria® 10 devices use the PR over
PCIe® solution to reconfigure the device, rather than
Configuration via Protocol (CvP) update. Partial reconfiguration allows you to reconfigure a
portion of the FPGA dynamically, while the remaining FPGA design continues to function. Create
multiple personas for a particular region in your design, without impacting operation in areas
outside this region. Partial reconfiguration enables the implementation of more complex FPGA
You can also include multiple parent and child partitions, or create
multiple levels of partitions in your design. This flow, referred to as the hierarchical
partial reconfiguration (HPR), includes a static region that instantiates the parent PR
region, and the parent PR region instantiating the corresponding child PR region. You can
perform the same PR region reprogramming for either the child or the parent partition.
Reprogramming a child PR region does not affect the parent or the static region. Reprogramming
the parent region reprograms the associated child region with the default child persona,
without affecting the static region.
Note: The HPR flow does not impose any restrictions on the number of
sub-partitions you can create in your design.
Partial reconfiguration provides the following advancements to a flat
Allows run-time design reconfiguration
Increases scalability of the design through time-multiplexing
Lowers cost and power consumption through efficient use of board
Supports dynamic time-multiplexing functions in the design
Improves initial programming time through smaller bitstreams
Reduces system down-time through line upgrades
Enables easy system update by allowing remote hardware change
The current version of the software introduces a new and simplified compilation flow for
The reference design consists of the following components:
wrapper for the reference design, connecting the board support package (BSP) subsystem to
the device pins.
bsp_top—top-level of the design that
contains all subsystems of the design. This module consists of three main sub-components -
PCIe® IP core, the DDR4 External Memory Interfaces
IP core, and the design top module. This layer of abstraction allows simulation of the
design top module through simulated Avalon-MM transactions.
design_core—core of the design that
handles generation of the PR region, the interface components such as clock crossing
Avalon-MM logic and pipeline logic, clocks, and the global reset.
The reference design creates a separate Altera IOPLL IP core-generated clock.
This clock creation decouples the PR logic clocking from both the
PCIe® clocking domain that runs at 250 MHz, and the
EMIF clocking domain that runs at 330 MHz. The clock for PR Logic is set at 250 MHz.
To ensure timing closure, modify the parameterization of the IOPLL IP core to a
lower clock frequency.
Figure 2. Timing Closure
Memory Address Mapping
PCIe® IP core connects to the design core
through two BARs (base address registers) - BAR 2 and BAR 4, which in turn connect
to their exclusive Avalon-MM interface.
In addition to connecting to the interface controls such as the Avalon-MM
freeze bridges and the PR region controller for the PR region, BAR 4 also connects
directly to up to 8 kB of memory in the PR region. Driver accessed components, such
as the PR IP core and the system description ROM use BAR 2. The following table
lists the memory address mapping for the
PCIe® Memory Address
System Description ROM
PR Region Controller
DDR4 Calibration Export
The reference design exports the status of the
DDR4 calibration, provided by the External Memory Interfaces IP core. Upon
initialization, the EMIF IP core performs training to reset the DDR4 interface. The
EMIF reports the success or the failure of the reset in its calibration flag. The
host decides the necessary action in the event of DDR4 failing the reset training,
so this interface is exported to the host.
The following table lists the memory
address mapping from the External Memory Interfaces IP core, to the PR logic:
The 2 GB DDR4 memory space is available to PR logic through an Avalon-MM
The floorplan constraints in your partial reconfiguration design physically
partitions the device.
This partitioning ensures that the resources
available to the PR region are the same for any persona that you implement.
To maximize the fabric amount available for the PR
region, the reference design constrains the static region to the smallest possible
area. This reference design contains two child PR regions of the parent PR
Figure 3. Reference Design Floorplan
Note: The child regions contained within the parent PR region can be of any
size. The small sizing of the child PR regions in the above figure is for
demonstration purposes only.
For more information about floorplanning, refer to Floorplan the Partial
Reconfiguration Design section in Volume 1 of the
Quartus® Prime Pro Edition Handbook.
This section describes the requirements and the procedure to run the reference
Hardware and Software Requirements
The reference design requires and uses the following hardware and software
Arria® 10 GX FPGA
development board with the DDR4 module connected to Hi-Lo interface
Linux Operating System - kernel version 3.10 or above
Super user access on the host machine
PCIe® slot to plug-in the
Arria® 10 GX FPGA development
Open source driver for this PR over
PCIe® reference design
Quartus® Prime Pro Edition software
Intel® FPGA Download Cable driver
The open source driver developed for this PR over
PCIe® reference design is tested using
The Linux driver accompanying this reference design is not a production
driver. You must adapt this driver based on your design.
Installing the Arria 10 GX FPGA Development Kit
For complete instructions on installing
and powering the
Arria® 10 GX FPGA development board in
your Linux system, refer to
Arria® 10 FPGA Development Kit User Guide.
Note: Before powering the board, set
the switch 4 (FACTORY) of the DIP switch bank (SW6)
to ON. Setting this switch to ON loads the factory image area of the flash memory
at boot time. Program the reference design into this factory image area. For
complete instructions on flashing the reference design onto the board, refer to
Bringing Up the Reference Design.
To access the reference design, navigate to the ref_designs sub-folder. Copy the
to the home directory in your Linux system.
To bring up the reference design on the board:
GX FPGA development board to an available
PCIe® slot in your host machine.
Connect the host machine's ATX auxiliary power connector to
the 12 V ATX input J4 of the development board.
Power-up the host machine.
Verify the micro-USB cable connection to the FPGA
development board. Ensure that no other applications that use the JTAG chain
Navigate to the
a10_pcie_devkit_cvp_hpr/software/installation folder in your
To overwrite the existing factory image on the board with
the reference design, execute the flash.pl script.
Pass the JTAG cable number of your connected device as an argument to the
script (for example, perl flash.pl 1).
Running this script configures the device with the
contents of the flash.pof file.
This parallel object file comes directly from the a10_pcie_devkit_cvp.sof file present in
the output_files directory. The
flash.pof file acts as the base
image for the reference design.
Note: Ensure successful compilation of the design before
running this script.
Power-cycle the host machine.
Reference Design Components
The reference design contains the following design components.
This Platform Designer system contains all the subsystems of this reference design.
The system comprises of three main components - the top-level design, the
PCIe® IP core, and the DDR4 External Memory Interfaces IP core. The
system connects to external pins through the
PCI Express IP core
Arria® 10 Hard IP for
PCI Express® IP core is a Gen3x8, with a 256-bit
interface and running at 250 MHz.
The following table provides information on the configuration fields of the
PCI Express® IP core that the reference design
uses that are different from the default settings:
PCI Express® IP Core Configuration
Avalon-MM with DMA
Gen3:x8, Interface: 256-bit, 250 MHz
RX buffer credit
allocation for received requests vs
register access (CRA) Avalon-MM slave
Registers - BAR2
32-bit non-prefetchable memory
Registers - BAR4
32-bit non-prefetchable memory
Device Identification Registers
Capabilities - Device
Debug, and Extension Options
Enable Arria 10 GX
FPGA Development Kit Connection
equalization far-end TX preset
Note: Instantiate the
PCI Express® IP core as part
of a Platform Designer system.
Arria 10 DDR4 External Memory Interfaces IP Core
The ddr4_emif logic includes the
Arria® 10 External Memory Interfaces IP core. This IP
core interfaces to the DDR4 external memory, with a 64-bit interface that runs at
1066.0 MHz. Also, the IP core provides 2 GB of DDR4 SDRAM memory space. The EMIF
Avalon-MM slave runs at 300 MHz clock.
The following table lists the configuration fields of the
Arria® 10 External Memory Interfaces IP core that are
different from the
Arria® 10 GX FPGA Development
Kit with DDR4 HILO preset settings:
Arria® 10 DDR4 External Memory Interfaces
DQ pins per DQS
Number of DQS
Alert# pin placement
I/O Lane with Address/Command Pins
Address/Command I/O lane of
Pin index of
This component forms the core of the design, and includes the following:
Partial Reconfiguration IP core
Clock crossing and pipe-lining for Avalon MM
System description ROM
Global Reset Logic
The PLL generates the main clock for this design. All logic,
excluding the pcie ip, pr
ip, and ddr4 emif run using this 250
MHz clock. The
PCIe® generates the global reset,
along with the PLL reset signal. On power up, a countdown timer, tcd2um counts down using the internal 50 MHz
oscillator, to a 830 μs delay. Until the timer reaches this delay, the PLL is held
in reset, deasserting the locked signal. This action freezes the design, and because
the PLL locked signal is ORed with the
reset, the design also is held in reset. Once the timer reaches 830 μs, the design
functions normally, and enters a known state.
PR Region Wrapper
The PR region wrapper contains the PR region controller, freeze bridges, and
the personas. The region controller interacts with the driver over the Avalon-MM
interface to initiate PR, and act as a bridge for communicating to the PR region for
freeze and start requests initiation.
Parent PR Region
The parent PR region contains two sets of PR region controllers and freeze bridges, because of
the two child personas within the parent PR region. Each set of PR region controller
and freeze bridge is dedicated to one child persona.
Arria 10 Partial Reconfiguration Region Controller IP Core
Arria® 10 Partial
Reconfiguration Region Controller IP core to initiate a freeze request to the PR
region. The PR region finalizes any actions, on freeze request acknowledgment. The
freeze bridges also intercept the Avalon-MM interfaces to the PR region, and
correctly responds to any transactions made to the PR region during partial
reconfiguration. Finally, on PR completion, the region controller issues a stop
request, allowing the region to acknowledge, and act accordingly. The fpga-region-controller program provided with this
reference design performs these functions.
The reference design configures the Partial Reconfiguration Region Controller
IP core to operate as an internal host. The design connects this IP core to the
PCI Express® IP core, via an instance of the
Avalon-MM interface. The PR IP core has a clock-to-data ratio of 1. Therefore, the
PR IP core is not capable of handling encrypted or compressed PR data.
The following table lists the configuration fields of the
Arria® 10 Hard IP for
PCI Express® IP core that are different from the preset settings:
Enable JTAG debug
Enable Avalon-MM slave
Input data width
Partial Reconfiguration Logic
The reference design provides the following personas:
Table 5. Reference Design Personas
Performs a sweep across a memory span, first
writing, and then reading each address.
Provides access to a 27x27 DSP multiplier,
and demonstrates hardware acceleration.
Includes a basic 32-bit unsigned adder, and
demonstrates hardware acceleration.
Game of Life
Includes an 8x8 Conway's Game of Life, and
demonstrates hardware acceleration.
A wrapper that instantiates two child partitions. The parent
persona also connects the two child personas to the static
region with their own PR region controller, BAR freeze
bridge, and DDR4 freeze bridge.
Each persona has an 8-bit persona_id field
in the pr_data register to indicate a unique
identification number. A 32-bit control register and 16 I/O registers follow the
8-bit persona_id. The 16 I/O registers are 32-bit
each, with 8 bits for device inputs, and 8 bits for device outputs. Each persona
uses these registers in different ways. For more information, refer to the source
code for each of the personas.
Additionally, the reference design provides a template to implement
your custom persona. This template persona allows you modify the RTL, create a
wrapper to interface with the register file, compile, and run your design.
Compiling the Reference Design
In order to compile the other personas in the
design, you must export the static region from the compiled base revision. Then, the
implementation revisions that define these personas import the static region. To
compile the HPR child personas, you must export the HPR parent region from the
compiled HPR parent revision. Then, the implementation revisions that define these
child personas import the HPR parent region.
To compile the base revision of the reference design, run the following command
from the project directory level:
This assignment imports the .qdb file
representing the reference design static region logic into the subsequent PR
persona implementation compile. Each implementation revision also contains
one or two ENTITY_REBINDING assignment.
This assignment links the hierarchy of the static region and the hierarchy
of the PR persona. For example, a10_pcie_devkit_cvp_ddr4_access.qsf contains the following
assignment imports the .qdb file
representing the HPR parent region into the subsequent into the subsequent HPR
child region compilation. Because the HPR child revisions comprise of two child
regions, they contain two ENTITY REBINDING
region offset value must be a hexadecimal number.
The PR region is reset during this operation.
Provides help for program_jtag_pcie script.
The example_host_uio module demonstrates the
FPGA device access. This application interacts with each persona, verifying the
contents and functionality of the personas.
The program requires a
PCIe® device number, followed by optional parameters
of the seed for generating test data, number of tests performed, and verbosity for
displaying extra information.
Table 8. example_host_uio Command-Line
Specifies the seed to use for number
generation. Default value is 1.
-d, --device [<device index>]
PCIe® device number. For example, -d 0000:03:00.0
Allows you to print additional information during
Allows you to specify the iterations for the test you wish to
Provides help for example_host application.
The reference design supports signal tapping the PR regions, through
hierarchical hubs. This feature facilitates the on-chip debugging of a specific
persona. The static region and the parent PR region contains the SLD agent, that
communicates with an SLD host. You must instantiate the SLD host in the persona
that you wish to signal tap. You must include the .stp file in the synthesis-only revision of a given persona. Do
not include the signal tap file in the base revision, or the .qsf file of other personas, if the .stp file is specific to a persona.
Follow these guidelines when signal tapping the PR logic:
Use pre-synthesis registers
An instance cannot cross partition boundaries
Do not trigger across partition boundaries
For complete information on the debug infrastructure using
hierarchical hubs, refer to Debugging Partial
Reconfiguration Designs Using Signal Tap
Logic Analyzer section in Volume 3 of the
Quartus® Prime Pro Edition handbook.
Your custom top-level entity must match the ports for the custom_persona module, defined in the source/templates/pr_logic_template.sv file.
The following example shows interfacing your design with the Avalon-MM
interface, controlled over PCIe register file:
module custom_persona #(
parameter REG_FILE_IO_SIZE = 8
input wire clk,
//active low reset, defined by hardware
input wire rst_n,
//Persona identification register, used by host in host program
output wire [31:0] persona_id,
//Host control register, used for control signals.
input wire [31:0] host_cntrl_register,
// 8 registers for host -> PR logic communication
input wire [31:0] host_pr [0:REG_FILE_IO_SIZE-1],
// 8 Registers for PR logic -> host communication
output wire [31:0] pr_host [0:REG_FILE_IO_SIZE-1]
Utilize any of the parallel I/O port (PIO) register
files for customization. The host_pr
register sends the data from the persona to the host machine. The
pr_host register sends the data
from the host machine to the persona.
In your top-level entity file, specify the persona ID as any
assign persona_id = 32'h0000_aeed;
Note: The example template uses only 8 bits, but you can
specify any value, up to 32 bits.
Set the unused output ports of the pr_host register to
//Tying unused output ports to zero.
for (i = 2; i < REG_FILE_IO_SIZE; i = i + 1) begin
assign pr_host [i] = 32'b0;
Modify your persona_impl_revision_name.qsf to include the following
Update the partial reconfiguration flow script to define
your new PR implementation.
Update the a10_pcie_devkit_cvp.qpf
project file to include your implementation
PROJECT_REVISION = "<persona_impl_revision_name>"
Compile the revision.
For complete information on adding
a custom persona to a PR design, refer to Adding a
New Persona to the Design section in the Partially Reconfiguring a Design on
Arria® 10 GX FPGA Development
Board application note.