Accelerator Funtional Unit Developer Guide for Intel FPGA Programmable Acceleration Card N3000

RSS

UG-20248 | 2019.12.06

Version Information

Updated for:
Intel® Acceleration Stack for Intel® Xeon® CPU with FPGAs 1.1

1. About This Document

This document serves as a high level guide for system architects and hardware developers in developing Accelerator Functional Units (AFUs) for the Intel FPGA Programmable Acceleration Card N3000.

This document is organized as follows:

Document Introduction and required background knowledge
High Level Description
Developing AFUs
Debugging AFUs

1.1. Acronym List

Acronym	Expansion	Description
Intel^® FPGA PAC	Intel FPGA Programmable Acceleration Card	Intel FPGA PAC N3000 is a full-duplex 100 Gbps in-system re-programmable acceleration card for multiworkload networking application acceleration.
AFU	Accelerator Functional Unit	Hardware Accelerator implemented in FPGA logic which offloads a computational operation for an application from the CPU to improve performance.
AF	Acceleration Function	Compiled Hardware Accelerator image implemented in FPGA logic that accelerates an application.
API	Application Programming Interface	A set of subroutine definitions, protocols, and tools for building software applications.
DPDK	Data Plane Development Kit	The Data Plane Development Kit consists of libraries to accelerate packet processing workloads running on many CPU architectures, including x86, POWER and ARM processors. DPDK runs mostly on Linux with a FreeBSD port available for a subset of DPDK features. The Open Source BSD License DPDK licenses DPDK.
FIU	FPGA Interface Unit	FIU is a platform interface layer that acts as a bridge between platform interfaces like PCIe* and AFU-side interfaces such as CCI-P.
OPAE	Open Programmable Acceleration Engine	The OPAE is a set of drivers, utilities, and API's for managing and accessing AFs.

2. Introduction

Before using this guide, refer to the Intel^® Acceleration Stack User Guide: Intel^® FPGA PAC N3000 , referred to as Intel^® FPGA PAC N3000 User Guide throughout this document. The Intel^® FPGA PAC N3000 User Guide provides an overview of the capabilities of the Intel^® FPGA PAC and provides instructions for installation and setup of hardware and software components of the stack, including the Open Programmable Acceleration Engine (OPAE) tools used in running diagnostic tools and remotely loading FPGA images. It is essential to familiarize yourself with the concepts developed and to complete the installation and setup procedures covered in the Intel^® FPGA PAC N3000 User Guide.

To perform AFU development, install the Intel^® Acceleration Stack for Development as described in the Intel^® FPGA PAC N3000 User Guide.

This guide for AFU development builds on the concepts and environment setup established in the Intel^® FPGA PAC N3000 User Guide.

2.1. Base Knowledge and Skills Prerequisites

The Intel^® Acceleration Stack is an advanced application of FPGA technology. The platform-level complexity has been abstracted away for the AFU developer by the inclusion of all interfaces in the FPGA factory image and a standard CCI-P interface to which you can design your AFU.

This guide assumes the following FPGA logic design-related knowledge and skills:

Familiarity with FPGA compilation flows, including the Intel^® Quartus^® Prime Pro Edition design flow.
Knowledge and skills in static timing closure, including familiarity with the Timing Analyzer tool in Intel^® Quartus^® Prime Pro Edition, applying timing constraints, Synopsys* Design Constraints (.sdc) language and Tcl scripting, and design methods to close on timing critical paths.
Knowledge and skills with industry standard RTL and coding practices to create logic that can be synthesized.
High level synthesis (HLS) and Platform Designer design entry tools are supported.
Knowledge and skill with industry standard RTL simulation tools.
Knowledge and skill with the Signal Tap Logic Analyzer tool in the Intel^® Quartus^® Prime Pro Edition software.

2.1.1. Considerations

If you are familiar with other Intel^® Acceleration Products, there are similarities and differences between the Intel^® Programmable Acceleration Card with Intel^® Arria^® 10 GX FPGA operation:

Similarities:
- CCI-P interface between user logic and Intel supplied PCIe interface
- OPAE kernel driver and tools for diagnostics and remote debugging
- FPGA region dedicated to OPAE management logic
Differences:
- Partial reconfiguration is not supported
  - The FPGA is a flat design loaded by on board flash
  - User must include encrypted blocks for board management
- ASE simulation is not supported
- Automated simulation and synthesis environment set up are not supported
Note: The OPAE version for the Intel^® FPGA PAC N3000 is not compatible, with previous and current versions of OPAE supporting Intel^® PAC with Intel^® Arria^® 10 GX FPGA.

3. High Level Description

The Intel^® FPGA PAC N3000 provides you with a rapid design methodology for creating complex FPGA and Intel^® Xeon^® networking applications. You are provided the following:

An Intel^® certified board with Intel^® Arria^® 10 GT FPGA, PCIe* interfaces, external memories, board management controller and Ethernet network interface devices.
FPGA factory image design demonstrating all interfaces.
Software tools for running board diagnostics with FPGA factory image, performing FPGA remote update, and reading board sensors.
Internal FPGA Nios^® II controller and firmware for Ethernet re-timer provisioning and board control functions. You must include this block in your design.
The FPGA design flow, which supports flexible Ethernet data flow configurations supporting your developed packet processing functions.

The Intel^® supplied board, FPGA IP blocks and software allow you to focus on your value added functionality.

3.1. Steps for Creating Your AFU

The following steps are suggested for designing a custom FPGA application for the Intel^® FPGA PAC N3000:

Become familiar with the board and FPGA block diagrams, interfaces and code provided within the Intel^® FPGA PAC N3000 factory image.
Review the Intel^® Acceleration Stack for Intel^® Xeon^® CPU with FPGAs Core Cache Interface (CCI-P) Reference Manual. You must follow the interface requirements and include required registers in your design for proper Intel^® FPGA PAC N3000 operation.
Define and plan your FPGA application.
Copy the Initial_Shell_AFU files and directory structure. This directory structure is the starting point for your design.
Implement your FPGA application. You can use one or a combination of the following design entry methods:
1. RTL (Verilog/VHDL)
2. Platform Designer
3. HLS
Note: Existing design blocks can be added as required.
Implement host software code.
Simulate your design at the unit level.
Create timing constraints files.
Update the Intel^® Quartus^® Prime Settings File (afu.qsf) to add your new blocks.
Compile, synthesize, place and route your new design using provided makefile.
Validate timing closure.
Validate power consumption.
The provided makefile compilation script includes a post-compilation script that creates a raw binary file.
The raw binary file is used an input to the Intel^® Acceleration Stack utility PACSign. PACSign adds a required header to the raw binary. The output file from PACSign is validated by the Intel^® FPGA PAC N3000 Intel^® MAX^® 10 Root of Trust for storage in the Intel^® Arria^® 10 flash. includes the utility PACSign. PACSign adds a header to the generated raw binary.
Flash the binary file produced by PACSign into FPGA flash using fpgasupdate.
Use the rsu utility to load the new FPGA from flash into the FPGA.
If needed, use the Signal Tap tool to diagnose and resolve issues.

3.2. Intel FPGA PAC N3000 Block Diagram

The board level Intel^® FPGA PAC N3000 block is shown below:

Figure 1. Intel^® FPGA PAC N3000 Block Diagram

As can be seen in Figure 1, the Intel^® Arria^® 10 FPGA is central to data and control flow. Within the Intel^® Arria^® 10 FPGA, there are both data and control IP cores that are required for the board to work properly. You must include these required IP cores in your designs. Figure 2 illustrates the Intel^® provided required and optional blocks as well as the ccip_std_afu block where your design is instantiated.

Figure 2. ccip_std_afu Block

3.2.1. In-Line Data Path

The Intel^® FPGA PAC N3000 supports multiple data path options. Your application can use one or more of these data options.

Ethernet data can be processed in-line where traffic traverses: QSFP > Intel^® Arria^® 10 FPGA > Intel^® Ethernet Controller XL710-BM2 NIC > Host and/or QSFP > Intel^® Arria^® 10 FPGA > Host. These data paths are shown below:

Figure 3. Data Path

Data can also be processed in a look-aside configuration where the data comes into the Intel^® Arria^® 10 FPGA from the host PCIe* interface, the FPGA processes the data and then sends the data back to the host through the PCIe* connection. Some examples of look-aside processing are compression/de-compression and encryption/decryption.

3.2.2. Supported Ethernet Network Configurations

The Intel^® FPGA PAC N3000 Intel^® FPGA PAC has three network configurations:

Network Configuration	QSFP28 A	QSFP28 B	Intel^® XL710 #1	Intel^® XL710 #2
8 x 10GbE	4 x 10GbE	4 x 10GbE	4 x 10GbE	4 x 10GbE
2 x 2 x 25GbE	2 x 25GbE	2 x 25GbE	2 x 40GbE	2 x 40GbE
4 x 25GbE	4 x 25GbE	Not Used	2 x 40GbE	2 x 40GbE

2 – QSFP ports where each QSFP supports 4 – 10 GbE lanes – this configuration is referred to as 8 X 10 G
2 – QSFP slots where each QSFP supports 2 – 25 GbE lanes – this configuration is referred to as 2 x 2 x 25 G
1 – QSFP port where 4 – 25 GbE lanes are supported. This configuration is referred to as 4 x 25 G

Note: The above network configurations are the only ones supported.

The fpga_top block contains a Nios^® II processor and firmware that configures the network settings for the Intel^® C827 Ethernet re-timer device. This Nios^® II firmware is not user editable.

The Intel^® Ethernet Controller XL710-BM2 network interface controller (NIC) is configured during board manufacturing to be either 10G or 40G. You cannot change the Intel^® Ethernet Controller XL710-BM2 NIC to switch between 10G and 40G. If your data path requires the Intel^® Ethernet Controller XL710-BM2 NIC, then you cannot switch between 10G and 25G network configurations. You can switch FPGA images between any of the supported 25G configurations.

3.2.3. Provided Files

The Intel^® FPGA PAC N3000 Acceleration Stack for Development software release provides the files for an example design. You can review this file set as a learning step for the creation of your design.

To access the files, go to your Intel^® FPGA PAC N3000 software installation directory and enter the following commands:

$ cd <N3000 Installation Directory>/inteldevstack/rtl/pac_n3000_rtl_1.3.6
$ export N3000_EXAMPLE_ROOT=$PWD

3.2.3.1. Directory Structure

The supplied FPGA files are a combination of clear text and encrypted files.

The directory structure of the supplied source files is shown below:

Directory Structure of the Factory_Image sub directory:

/hw/afu – this is where the AFU factory image example is located
- /hw - Sub directory with clear text RTL, afu.qsf and afu.sdc
- /sw - Sub directory with example software code
/hw/pac – this is where Ethernet MAC, external memory interface, and encrypted FIM is included
/prj/pac_baseline – Intel^® Quartus^® Prime project files
/prj/pac_baseline/build – programming files and build reports after compilation completes
At the top of the directory tree is the Makefile used in compiling the project.
The hello_afu sub directory contains a simple example AFU illustrating design points. The Initial_Shell_AFU contains a starting directory structure for your new AFU design.

3.2.4. Internal Interfaces

The ccip_std_afu module has the following interfaces:

Core Cache Interface (CCI-P) – This is an FPGA – Host PCIe* interface required for OPAE stack operation.
Ethernet interface – This interface provides each Ethernet interface as individual or Multiplexed Avalon^® streaming interface bus or buses.
Local Memory – Each external memory has an Avalon^® memory-mapped interface interface.
PCIe* – Optional secondary PCIe* interface can be included.

3.2.4.1. Core Cache Interface (CCI-P)

The Intel^® FPGA PAC N3000 uses the CCI-P interface for compatibility with the OPAE software stack and drivers. The Intel^® FPGA PAC N3000 has the FIU capabilities of the Intel^® PAC with Intel^® Arria^® 10 GX FPGA as shown in the Comparison of FIU Capabilities section of the Intel^® Acceleration Stack for Intel^® Xeon^® CPU with FPGAs Core Cache Interface (CCI-P) Reference Manual.

Note: You must develop a detailed understanding of the CCI-P Interface as described in the CCI-P Interface section of the Intel^® Acceleration Stack for Intel^® Xeon^® CPU with FPGAs Core Cache Interface (CCI-P) Reference Manual.

The Intel^® FPGA PAC N3000 has the signals in the CCI-P interface as follows:

Signal	Width	Direction	Description
`pClk`	1	Input	200 MHz system clock. All CCI-P signals are synchronous to this signal.
`pClkDiv2`	1	Input	200 MHz system clock. This signal is a copy of `pClk`. Please ignore name
`pClkDiv4`	1	Input	200 MHz system clock. This signal is a copy of `pClk`. Please ignore name.
`uClk_usr`	1	Input	User clock – Default = 312.5 MHz clock. You cannot change this clock; however, if a custom clock is needed, you have to instantiate your own PLL inside the AFU using the global `G_CLK100` as a reference clock.
`uClk_usrDiv2`	1	Input	User clock – Default = 156.25 MHz clock. You cannot change this clock; however, if a custom clock is needed, you have to instantiate your own PLL inside the AFU using the global `G_CLK100` as a reference clock. This clock is not required to be uClk_usr divided by 2
`G_CLK100`	1	Input	100 MHz global reference clock, for optional PCIe* IP core or additional PLLs if needed
`t_if_ccip_Rx`	struc	Input	CCI-P data input structure defined in ccip_if_pkg.sv
`t_if_ccip_Tx`	struc	Output	CCI-P data output structure defined in ccip_if_pkg.sv
`pck_cp2af_softReset`	1	Input	Active high reset. Synchronous with `pClk` asserted for 256 clock cycles.
`pck_cp2af_pwrState`	2	Input	Present, but not used
`pck_cp2af_error`	1	Input	Present, but not used

3.2.4.1.1. FPGA Internal Register Access

Access to internal FPGA registers with the PCIe* 0 CCI-P interface uses MMIO access. You may use the following types of internal registers:

Direct access
Indirect access

Direct access registers consist of MMIO addressable registers. Examples of these registers includes – Bitstream ID, error status, and Single Event Upset EMR status registers.

The CCI-P protocol MMIO address space is limited to 256 kB. The indirect access registers provide a mechanism to address larger areas by including control and response registers for extended slave addressing. AFU designers may use the provided ccip_to_avmm module to provide indirect access for your Avalon^® memory-mapped interface slave modules. The ccip_to_avmm module block diagram is shown below:

Figure 4. CCI-P to Avalon^® memory-mapped interface Block

As can be seen in this block diagram, this module consists of the following:

ccip_async_shim – CCI-P to and from Avalon^® clock domain crossing
ccip_avmm_mmio – converts MMIO to and from Avalon^® memory-mapped interface
bbs_regs_mm_wrap – contains CCI-P required DFH and AFU ID registers and indirect command and status registers

The indirect command and status registers are defined as follows:

Table 1. Control Register
Field Name	Range	Access	Description
`cmd`	`[63:62]`	`RW`	Command for slave: 0x0 – NOP 0x1 – indirect read request 0x2 – indirect write request
`addr`	`[61:32]`	`RW`	Slave address
`Write data`	`[31:0]`	`RW`	Slave write data

For an indirect write request:

Write the following to the Control register:
- cmd = 0x2
- addr
- Write data
Poll on the RW valid field of the Status register for RW valid = 1 to verify that the write is successful.

For an indirect read request,:

Write the following to the Control register::
- cmd = 0x1
- addr
Poll on the RW valid field of the Status register for RW valid = 1 to verify that the RD Data field contains valid data.

`BBS_regs_mm_wrap` Access Behavior

The following figures show the bbs_regs_mm_wrap upstream Avalon^® memory-mapped interface slave to downstream Avalon^® memory-mapped interface slave waveforms for indirect write and read operations.

Note: Pay attention to the downstream Avalon^® memory-mapped interface master waveforms for proper operation with your slave module.

Indirect Write and Read Requests with Non-Blocking Access

The back pressure signal (avmm_s_waitrequest) is not used from the indirect access module to the CCI-P; and a write (WR) or read (RD) transaction can start at any time, but must complete in the next clock cycle.

Figure 5 and Figure 6 demonstrate this behavior:

Figure 5. Avalon^® memory-mapped interface Waveforms for Indirect Write Request with NONBLOCKING_ACCESS_EN = 1

Figure 6. Avalon^® memory-mapped interface Waveforms of Indirect Read Request with NONBLOCKING_ACCESS_EN = 1

Write and Indirect Read Requests with Blocking Access

The back pressure signal (avmm_s_waitrequest) is always set to ‘1’ in the NOP state; and a write (WR) or read (RD) transaction can start when avmm_s_waitrequest = 1, but cannot finish until avmm_s_waitrequest != 0.

Figure 7 and Figure 8 demonstrate this behavior:

Figure 7. Write Request with NONBLOCKING_ACCESS_EN = 0

Figure 8. Waveform of Indirect Read Request with NONBLOCKING_ACCESS_EN = 0

Avalon^® memory-mapped interface Master Response to a Read Request with Non-Blocking and Blocking Access

You can read data on the bus for the following conditions:

When AVMM_MASTER_READDATAVALID_EN = 1 and avmm_m_readdatavalid are valid
When AVMM_MASTER_READDATAVALID_EN = 0 and (!avmm_m_waitrequest & avmm_m_read) are valid

Figure 9 and Figure 10 demonstrate this behavior:

Figure 9. Avalon^® memory-mapped interface Master Response to a Read Request with NONBLOCKING_ACCESS_EN = 1

Figure 10. Avalon^® memory-mapped interface Master Response to a Read Request with NONBLOCKING_ACCESS_EN = 0

3.2.4.2. Ethernet Interface

The Intel^® FPGA PAC N3000 has Ethernet MAC IP cores to provide Ethernet receive packet delineation and transmit packet origination. The Ethernet MACs are instantiated in both the network and the Intel^® FPGA PAC N3000 Intel^® Ethernet Controller XL710-BM2 NIC, as shown below:

Figure 11. Instantiated Ethernet MACs

The nfv_eth_wrapper module is configurable by Verilog parameters for the type of Ethernet interface (10, 25 or 40 G), number of interfaces and aggregated or disaggregated style of the AFU interface. The setting of these Verilog parameters is performed by the Makefile option settings described in the Build with make section. The nfv_eth_wrapper module includes the following:

Ethernet MAC
PLL
Multiplex/De-Multiplex blocks

The AFU Ethernet interface has three options with the following properties:

Aggregated:

One Avalon^® streaming interface bus aggregating all traffic from each Ethernet interface
Common clock
Each Ethernet channel is identified by Avalon^® streaming interface channel identifier
Full Ethernet MAC statistics provided

The aggregated option allows your AFU to have a common packet processing pipeline. The aggregated option uses more FPGA resources and introduces delay from a packet buffer.

Figure 12. 8x10G Multiplexor

Figure 13. 8x10 De-Multiplexor

Disaggregated:

Each Ethernet MAC has an Avalon^® streaming interface bus provided as an array of Avalon^® streaming interface. Each channel is identified by array index.
Received packets with errors (CRC, length errors) are dropped from MAC. The AFU logic must control whether these packets with errors are dropped.
Egress FIFO saturation based flow control is provided to AFU.
One common clock is used by AFU logic.

The disaggregated configuration reduces FPGA resources removing the multiplex/de-multiplex blocks.

Figure 14. MAC RX Packet MUX

Figure 15. MAC TX DEMUX

Lightweight Mode

Disaggregated Avalon^® streaming interface interfaces from each MAC.
MUX function passes received traffic directly to the AFU.
The AFU must control the data stream by removing frames with errors and controlling the flow.
Each MAC interface has a separate clock.
No Ethernet statistics provided in Ethernet MACs.

Note: The Lightweight mode is not supported for 10G applications.

Figure 16. MAC RX Packet MUX

Figure 17. MAC TX DEMUX

3.2.4.3. Ethernet MAC

The 25 GbE MAC IP core is documented in the 25G Ethernet Intel^® Arria^® 10 FPGA IP User Guide. The Intel^® FPGA PAC N3000 configures the 25 GbE MAC with the following parameters set with the following parameters:

Table 2. 25G MAC IP Setting
Parameter	IP Core parameter setting
Ready Latency	0
Enable RS-FEC	Off
Enable flow control	On
Enable link fault generation	On
Enable preamble pass through	Off
Enable TX CRC pass through	Off
Enable MAC statistics counters	On
Enable IEEE 1588	Off

The Intel C827 Re-timer performs FEC functionality, therefore the A10 Ethernet MAC does not have RS-FEC enabled.

The 10 GbE MAC IP core is documented in: Low Latency Ethernet 10G MAC Intel^® Arria^® 10 FPGA IP Design Example User Guide

The 40 GbE MAC IP core is documented in: Low Latency 40-Gbps Ethernet IP Core User Guide

The 40 and 10 GbE MAC IP core are set with the following parameters:

Table 3. 40G MAC IP Setting
Parameter	IP core parameter setting
Enable SyncE	Off
PHY reference	644.53125MHz
Use external TX MAC PLL	On
Flow control mode	Standard flow control
Average inter-packet gap	12
Enable 1588 PTP	Off
Enable link fault generation	On
Enable TX CRC insertion	On
Enable preamble pass through	Off
Enable alignment EOP on FCS word	On
Enable TX statistics	On
Enable RX statistics	On
Enable strict SFD checking	Off

3.2.4.4. 40G – 25G Gearbox

For 25 GbE operation, the Intel^® Arria^® 10 FPGA provides a gearbox that rate adjusts between the 25 GbE network interface and the Intel^® Ethernet Controller XL710-BM2 NIC 40 GbE interface.

Received 25 GbE traffic is written into a per port 32 kB Intel^® Arria^® 10 FPGA FIFO. The FIFO data is read out on packet boundaries using a 40 GbE rate where an entire packet is transferred to the Intel^® Ethernet Controller XL710-BM2 NIC. The Intel^® Arria^® 10 FPGA extends the interframe packet gap to the Intel^® Ethernet Controller XL710-BM2 NIC such that the data rate is 40 Gb, however the number of packets transferred is determined by the number of packets received from the 25 GbE network port.

The Intel^® Ethernet Controller XL710-BM2 NIC sends Ethernet traffic to the Intel^® Arria^® 10 FPGA over a 40 GbE path. The Intel^® Arria^® 10 FPGA buffers the 40 GbE traffic in a 32 kB packet-based FIFO. If the Intel^® Arria^® 10 FPGA FIFO exceeds half fill level, then the Intel^® Arria^® 10 FPGA asserts a backpressure external pin, signaling the Intel^® Ethernet Controller XL710-BM2 NIC to extend the interframe packet gap. Once the Intel^® Arria^® 10 FPGA FIFO capacity drops to a quarter of capacity, then the backpressure external pin is de-asserted. This extended interframe packet gap reduces the packet rate such that the resulting data rate is 25 Gb. The backpressure signals are connected to the ccip_std_afu module. The Intel^® Ethernet Controller XL710-BM2 NIC to Intel^® Arria^® 10 FPGA data flow is shown in this figure:

Figure 18. MAC TX DEMUX

3.2.4.5. External Memory Interfaces

The Intel^® FPGA PAC N3000 has the following external memory interfaces as shown in the board block diagram below:

Figure 19. External Memory Interfaces

DDR4 – 2133 Mb/s – total 9 GB
- DDR4A and DDR4B - each 4 GB banks
  - 64-bit wide
  - Ping-Pong physical interface
- DDR4C - 1 GB bank
  - 16-bit wide
QDR4 – 1066 MHz – 144 Mb
- 8M x 18

DDR4A and DDR4B

Both DDR4A and DDR4B use the Ping Pong PHY described in the Intel^® Arria^® 10 EMIF Ping Pong PHY Description section of the External Memory Interfaces Intel^® Arria^® 10 FPGA IP User Guide.

The Ping Pong PHY is physically implemented in the board design. The Ping Pong PHY design has two independent memory controllers per DDR4 interface where your interface consists of two Avalon^® memory-mapped interface interfaces. See DDR4A user interface below (please note, DDR4B is identical).

ccip_std_afu Direction	Width	Signal Name	Description
input		ddr4a_avmm_0_clk	266 MHz clock sourced from EMIF
input		ddr4a_avmm_0_reset_n	Active low reset to user logic. Reset for the user clock domain. Asynchronous assertion and synchronous de-assertion
input		ddr4a_avmm_0_waitrequest	Wait-request is asserted when controller Avalon^® memory-mapped interface interface is busy
input	[255:0]	ddr4a_avmm_0_readdata	Read data from external memory
input		ddr4a_avmm_0_readdatavalid	Indicates readdata is valid when high
output	[6:0]	ddr4a_avmm_0_burstcount	Number of transfers in each read/write burst
output	[255:0]	ddr4a_avmm_0_writedata	AFU supplied data to written to external memory
output	[25:0]	ddr4a_avmm_0_address	Word address for Avalon^® memory-mapped interface interface of memory controller
output		ddr4a_avmm_0_write	Write request from AFU
output		ddr4a_avmm_0_read	Read request from AFU
output	[31:0]	ddr4a_avmm_0_byteenable	Write byte enable from AFU
DDR4A_ 1 Interface
input		ddr4a_avmm_1_clk	Copy of ddr4a_avmm_0_clk
input		ddr4a_avmm_1_reset_n	Secondary active low reset to user logic. Reset for the user clock domain. Asynchronous assertion and synchronous de-assertion
input		ddr4a_avmm_1_waitrequest	Wait-request is asserted when controller Avalon^® memory-mapped interface interface is busy
input	[255:0]	ddr4a_avmm_1_readdata	Read data from external memory
input		ddr4a_avmm_1_readdatavalid	Indicates readdata is valid when high
output	[6:0]	ddr4a_avmm_1_burstcount	Number of transfers in each read/write burst
output	[255:0]	ddr4a_avmm_1_writedata	AFU supplied data to written to external memory
output	[25:0]	ddr4a_avmm_1_address	Word address for Avalon^® memory-mapped interface interface of memory controller
output		ddr4a_avmm_1_write	Write request from AFU
output		ddr4a_avmm_1_read	Read request from AFU
output	[31:0]	ddr4a_avmm_1_byteenable	Write byte enable from AFU

You can combine both of the Ping Pong Avalon^® memory-mapped interface interfaces from one DDR4 bank to form a 512-bit interface with an Avalon^® combiner. The factory image example demonstrates the use of the Avalon^® combiner.

The DDR4A and DDR4B interfaces are suited to large record storage, off chip deep packet queues and other storage needs.

DDR4C

The ccip_std_afu interfaces to DDR4C by an Avalon^® memory-mapped interface interface as defined below:

ccip_std_afu Direction	Width	Signal Name	Description
input		ddr4c_avmm_0_clk	266 MHz clock sourced from EMIF
input		ddr4c_avmm_0_reset_n	Active low reset to user logic. Reset for the user clock domain. Asynchronous assertion and synchronous de-assertion
input		ddr4c_avmm_0_waitrequest	Wait-request is asserted when controller Avalon^® memory-mapped interface interface is busy
input	[127:0]	ddr4c_avmm_0_readdata	Read data from external memory
input		ddr4c_avmm_0_readdatavalid	Indicates readdata is valid when high
output	[6:0]	ddr4c_avmm_0_burstcount	Number of transfers in each read/write burst
output	[127:0]	ddr4c_avmm_0_writedata	AFU supplied data to written to external memory
output	[25:0]	ddr4c_avmm_0_address	Word address for Avalon^® memory-mapped interface interface of memory controller
output		ddr4c_avmm_0_write	Write request from AFU
output		ddr4c_avmm_0_read	Read request from AFU
output	[15:0]	ddr4c_avmm_0_byteenable	Write byte enable from AFU

QDR4 Interface

The external QDR4 SRAM is well suited to fast table look ups and external statistics counter storage due to the fast random access capabilities of QDR4 SRAM. QDR4 SRAM transfers 4 data words per clock cycle. QDR4 SRAM also has two independent bidirectional double data rate ports that support concurrent read/write transactions on both ports.

The multiple access ports of the QDR4 SRAM results in the internal interface providing 8 – Avalon^® memory-mapped interface interfaces for this external memory device. The interface is shown below:

ccip_std_afu Direction	Width	Signal Name	Description
input		qdr_avmm_clk	266 MHz clock sourced from EMIF
input		qdr_avmm_reset_n	Active low reset to user logic. Reset for the user clock domain. Asynchronous assertion and synchronous de-assertion
input		qdr_avmm_waitrequest [7:0]	Wait-request is asserted when controller Avalon^® memory-mapped interface interface is busy
input	[35:0]	qdr_avmm_readdata [7:0]	Read data from external memory
input		qdr_avmm_readdatavalid [7:0]	Indicates readdata is valid when high
output	[2:0]	qdr_avmm_burstcount [7:0]	Number of transfers in each read/write burst
output	[35:0]	qdr_avmm_writedata [7:0]	AFU supplied data to written to external memory
output	[21:0]	qdr_avmm_address [7:0]	Word address for Avalon^® memory-mapped interface interface of memory controller
output		qdr_avmm_write [7:0]	Write request from AFU
output		qdr_avmm_read [7:0]	Read request from AFU

3.2.4.6. Ethernet MAC Wrapper Register Access

Host processor access to Ethernet MAC Wrapper is by the CCI-P interface using MMIO indirect access as described in the FPGA Internal Register Access section. The RTL modules for register access are included in the encrypted portion of Intel^® FPGA PAC N3000 design and these modules must be included in your design.

There are two Ethernet MAC Wrappers where one wrapper is connected to the network and the other is connected to the Intel^® Ethernet Controller XL710-BM2 NIC, as shown below:

Figure 20. Ethernet Wrapper Register Access

The Ethernet MAC Wrapper registers consist of the following:

CCI-P required Device Feature Header (DFH) and Information registers are located inside fpga_top.
Indirect access control register and status registers are located in phy_indir_wrap.
Ethernet MAC, PHY and multiplex/de-multiplex control and status registers are located in nfv_eth_wrapper.

For an example of how software accesses the Ethernet MAC wrapper, see the python source file included with the OPAE software release installation:

inteldevstack/src/opae-*.*/usr/tools/extra/fpgadiag/fpgastats.py

The following description of registers below is provided for informational purposes. Do not change or modify this area code, but understanding how this works helps you create your AFU. When the lightweight mode is used, the Ethernet MAC registers are not included.

These registers are organized as follows:

Table 4. Ethernet MAC Registers
Register	Address Offset
ETH_GROUP_0_DFH	0x7000
ETH_GROUP_0_INFO	0x7008
ETH_GROUP_0_CTRL	0x7010
ETH_GROUP_0_STAT	0x7018
ETH_GROUP_1_DFH	0x8000
ETH_GROUP_1_INFO	0x8008
ETH_GROUP_1_CTRL	0x8010
ETH_GROUP_1_STAT	0x8018

The Information register consists of the following fields:

Table 5. Information Register Fields
FIELD NAME	RANGE	ACCESS	DEFAULT	DESCRIPTION
Reserved	[63:26]	RsvdZ	0x0	Reserved
MAC light weight mode	[25]	RO	0x0	0 - MACs are in normal mode 1 - MACs are in light weight mode
Direction	24	RO	0x0	0 – XL710 side 1 - Network side
Speed_Gbs	[23:16]	RO	0xA	Allowed: 10, 25, 40 Gbs.
NofPHYs	[15:8]	RO	0x8	Number of PHYs in group
GroupID	[7:0]	RO	0x0	Unique identifier of phy group. ETH_GROUP_0 = 0, ETH_GROUP_1 = 1

The indirect control field has one version for 10G mode and a second version for 25G and 40G mode. The 10G mode is shown below:

Table 6. 10G Indirect Control Field
FIELD NAME	RANGE	ACCESS	DEFAULT	DESCRIPTION
command	[63:62]	RW	0x0	Command: 0x0 - NOP 0x1 - RD request 0x2 - WR request
reserved	[61:54]	RO	0x0
device select	[53:49]	RW	0x0	0x0 - Ethernet Wrapper regs select 0x2, 0x4, 0x6, 0x8, 0xA, 0xC, 0xE, 0x10 - PHY select 0x3, 0x5, 0x7, 0x9, 0xB, 0xD, 0xF, 0x11 - MAC select
PHY select				device select = 0x2, 0x4, 0x6, 0x8, 0xA, 0xC, 0xE, 0x10
add features select	[48]	RW	0x0	0x0 - phy select 0x1 - reset controller / link status select
PHY Address/reset ctrl/link status	[47:32]	RW	0x0	add features select = 0x0: PHY reconfiguration interface www.altera.com/literature/hb/arria-10/ug_arria10_xcvr_phy.pdf add features select = 0x1: ref. to add features tab.
MAC register address	[48:32]	RW	0x0	When device select = 0x3, 0x5, 0x7, 0x9, 0xB, 0xD, 0xF, 0x11 This field is for Ethernet MAC IP registers as defined in: www.altera.com/en_US/pdfs/literature/ug/ug_32b_10g_ethernet_mac.pdf.
ethernet wrapper regs address	[48:32]			When device select = 0x0 This field includes Ethernet Mux/De-Mux registers
write data	[31:0]	RW	0x0	Write data for phy registers

For 25G and 40G mode:

Table 7. 25G and 40G Indirect Control Field
FIELD NAME	RANGE	ACCESS	DEFAULT	DESCRIPTION
command	[63:62]	RW	0x0	Command: 0x0 - NOP 0x1 - RD request 0x2 - WR request
reserved	[61:54]	RO	0x0
device select	[53:49]	RW	0x0	0x0 - ethernet wrapper regs select 0x2, 0x4, 0x6, 0x8 - PHY select 0x3, 0x5, 0x7, 0x9 - MAC select
PHY select				device select = 0x2, 0x4, 0x6, 0x8
add features select	[48]	RW	0x0	0x0 - phy select 0x1 - reset controller / link status select
PHY Address/reset ctrl/link status	[47:32]	RW	0x0	add features select = 0x0: add features select = 0x1: ref. to add features tab.
MAC select				device select = 0x3, 0x5, 0x7, 0x9
	[48:32]
Ethernet Wrapper regs address	[48:32]			When device select = 0x0 This field includes Ethernet Mux/De-Mux registers

3.3. Factory Image Description

The Intel^® FPGA PAC N3000 provides an example design that demonstrates usage of the key interfaces available to the ccip_std_afu module.

The block diagram of each network configuration is shown below:

Figure 21. Factory Image Block Diagram for 8x10 GbE

Figure 22. Factory Image Block Diagram for 2x2x25GbE

Figure 23. Factory Image Block Diagram for 4x25 GbE

4. Creating an Intel FPGA PAC N3000 FPGA Design

In this section, steps are provided to create your AFU. The following AFU design directories are included:

hello_afu – this is a simple AFU design illustrating basic design concepts
Factory_Image – this is a complex AFU design illustrating usage of Ethernet and external memories
Initial_Shell_AFU – this design directory serves as the starting point for your AFU. The required project files are included.

In this section, you are referred to these design directories as a way to highlight points in the AFU creation, compilation and running of an application on the Intel^® FPGA PAC N3000. Use these points to create more complex AFU designs for your specific application.

4.1. Create New Project Directory

The Initial_Shell_ provides the starting structure for your design.

Create a new project directory and copy the Initial_Shell_ files to the new project directory.

$ mkdir <Your new project directory name>
$ cd <Your new project directory name>
$ cp -R $N3000_EXAMPLE_ROOT/Initial_Shell_AFU/* .

Your design directory is now ready for your new design work.

4.2. Create Your AFU Design Files

As a minimum, create the following files for your AFU design:

ccip_std_afu.sv – this file is where your AFU connects to CCI-P fabric, external memory and Ethernet
An AFU file. You can see AFU examples in hello_afu/hw/afu/hw/rtl/hello_afu.sv and Factory_Image/hw/afu/rtl/afu_dma.sv
afu.qsf - this Intel^® Quartus^® Prime file adds your RTL and design files
afu.sdc – this Intel^® Quartus^® Prime file specifies your AFU timing constraints

4.2.1. ccip_std_afu.sv

The Initial_Shell_AFU includes ccip_std_afu.sv starting file where you can instantiate your AFU. Review of this file shows import ccip_if_pkg::* to include definitions of CCI-P interface structures.

import ccip_if_pkg::*;  //required for CCI-P definitions
module ccip_std_afu #( …..
;
;

There is a CCI-P interface register to improve timing as shown below:

    // =============================================================
    // Register SR <--> PR signals at interface before consuming it
    // =============================================================

    (* noprune *) logic [1:0]   pck_cp2af_pwrState_T1;
    (* noprune *) logic         pck_cp2af_error_T1;

    logic                       pck_cp2af_softReset_T1;
    t_if_ccip_Rx                pck_cp2af_sRx_T1;
    t_if_ccip_Tx                pck_af2cp_sTx_T0;

    // ============================================================= 
    // Register PR <--> PR signals near interface before consuming it
    // ============================================================= 
                                                     
    ccip_interface_reg inst_green_ccip_interface_reg  (              
        .pClk                           (pClk),                      
        .pck_cp2af_softReset_T0         (pck_cp2af_softReset),       
        .pck_cp2af_pwrState_T0          (pck_cp2af_pwrState),        
        .pck_cp2af_error_T0             (pck_cp2af_error),           
        .pck_cp2af_sRx_T0               (pck_cp2af_sRx),             
        .pck_af2cp_sTx_T0               (pck_af2cp_sTx_T0),          
                                                         
        .pck_cp2af_softReset_T1         (pck_cp2af_softReset_T1),    
        .pck_cp2af_pwrState_T1          (pck_cp2af_pwrState_T1),     
        .pck_cp2af_error_T1             (pck_cp2af_error_T1),        
        .pck_cp2af_sRx_T1               (pck_cp2af_sRx_T1),          
        .pck_af2cp_sTx_T1               (pck_af2cp_sTx)              
    );

Your AFU design connects to this registered CCI-P interface.

4.2.2. AFU File

Your AFU requires a CCI-P package and a UUID for proper connectivity with host software. See example below:

import ccip_if_pkg::*;
module hello_afu
   (
    input  clk,    // Core clock. CCI interface is synchronous to this clock.
    input  reset,  // CCI interface ACTIVE HIGH reset.
    // CCI-P signals
    input  t_if_ccip_Rx cp2af_sRxPort,
    output t_if_ccip_Tx af2cp_sTxPort
    );
	`define AFU_ACCEL_UUID 128'h850adcc2_6ceb_4b22_9722_d43375b61c66
    // The AFU must respond with its AFU ID in response to MMIO reads of
    // the CCI-P device feature header (DFH).  The AFU ID is a unique ID
    // for a given program.  Here we generated one with the "uuidgen"
    // program and stored it in the AFU's JSON file.  ASE and synthesis
    // setup scripts automatically invoke the OPAE afu_json_mgr script
    // to extract the UUID into afu_json_info.vh.
    logic [127:0] afu_id = `AFU_ACCEL_UUID;

See Factory_Image/hw/afu/rtl/afu_dma.sv for a more complicated example where multiple sub-AFUs are instantiated.

4.2.3. QSF File

The afu.qsf is where you include your AFU RTL and any other required implementation files. The Intial_Shell_AFU includes an afu.qsf file where you can add your specific files.

4.2.4. SDC File

The afu.sdc is where you include your AFU timing constraints files.

4.3. Build with make

The process of creating an Intel^® FPGA PAC N3000 FPGA image is simplified with the provided Makefile that automates the setting of compile parameters and combining your design files with the supplied source files. The Makefile flow starts with your design input files and ends after Intel^® Quartus^® Prime synthesizes and places the output and a binary FPGA file that is ready for secure signing with PACSign.

The make flow is only supported on Linux* platforms. Your development system requires the following:

Make:

make 3.81 (or newer)

Python:

Python 3.6

You invoke the make flow with the following command input syntax:

make [target] [options] [paths] [versioning]

Target—required, input only one:

2x2x25G
4x25G
8x10G
reopen—require a previously compiled design
archive—require a previously compiled design

Options

Option	Value	Description	Required	Default	Comment
GUI	0	run selected stage	NO	0
GUI	1	open Intel^® Quartus^® Prime GUI	NO	0
SEED	0 - 232-1	fitter seed	NO	1
STAGE	compile	execute full flow (step-by-step)	YES	Not applicable
	synthesis	execute analysis and synthesis			calls ipgenerate
	fitter	execute fitting (Fit, Place, Route)			calls ipgenerate
	fitter-timing	execute fitter and timing analysis			calls ipgenerate
	analysis-timing	execute timing analysis			requires completed fitter
	analysis-power	execute power analysis			requires completed fitter
	assembler	execute assembler			requires completed fitter
	ipgenerate	generate IPs
	dummy	do nothing			GUI only
USE_BBS_CLK	0	do not take user clock from BBS	NO	0
USE_BBS_CLK	1	take user clock from BBS	NO	0
INCLUDE_DIAGNOSTICS	0	exclude AFU diagnostics	NO	1
INCLUDE_DIAGNOSTICS	1	include AFU diagnostics	NO	1
INCLUDE_AFU_PCIE1	0	exclude AFU PCIe* 1	NO	0
INCLUDE_AFU_PCIE1	1	include AFU PCIe* 1	NO	0
INCLUDE_MEMORY	0	exclude all EMIFs	NO	1
INCLUDE_MEMORY	1	include all EMIFs	NO	1
INCLUDE_DDR4_A	0	exclude DDR4 A	NO	INCLUDE_MEMORY
INCLUDE_DDR4_A	1	include DDR4 A	NO	INCLUDE_MEMORY
INCLUDE_DDR4_B	0	exclude DDR4 B	NO	INCLUDE_MEMORY
INCLUDE_DDR4_B	1	include DDR4 B	NO	INCLUDE_MEMORY
INCLUDE_DDR4_C	0	exclude DDR4 C	NO	INCLUDE_MEMORY
INCLUDE_DDR4_C	1	include DDR4 C	NO	INCLUDE_MEMORY
INCLUDE_QDR	0	exclude QDR	NO	INCLUDE_MEMORY
INCLUDE_QDR	1	include QDR	NO	INCLUDE_MEMORY
MAC_LIGHTWEIGHT_MODE	0	disabled	NO	0	Disabled non required features in MACs for less logic consumption
MAC_LIGHTWEIGHT_MODE	1	enabled	NO	0
DATAPATH_MODE	normal	normal mode	NO	normal
	disaggregated	disaggregated channels mode
	lightweight	lightweight mode
INCLUDE_SEU	0	Excludes SEU detection circuit	NO	1
INCLUDE_SEU	1	Includes SEU detection circuit	NO	1
ARCHIVE_NAME	[any string]	.qar archive name for .qdb archive	NO	snapshot.qar	used by archive target

Usage Examples:

1. Set up your shell environment to use Intel^® FPGA PAC N3000 provided Intel^® Quartus^® Prime

$ source ~/inteldevstack/bin/init_env.sh

This example runs the full compile process of the 2 x 2 x 25 GbE factory image using the command line (non-GUI) mode with fitter seed equal to 5:

$ make 2x2x25G SEED=5 STAGE=compile

Note: Some designs may require multiple seed passes using seeds 1 to 8 to achieve timing closure. Also, the Design Space Explorer does not work with the Intel^® FPGA PAC N3000 make design flow.

Paths

Path	Description	Default
PROJECT_FILE	main project qpf	prj/pac_baseline/chip.qpf
PAC_ROOT	Intel^® FPGA PAC N3000 sources root where main .qip is located	hw/pac
AFU_ROOT	AFU sources root where afu.qsf is located	hw/afu/hw

Versioning

Versioning	Description	Default
`PAC_VER_MAJOR`	SemVer Major. 0 - 15	0
`PAC_VER_MINOR`	SemVer Minor. 0 – 255	0
`PAC_VER_PATCH`	SemVer Patch. 0 – 15	0
`REVISION_ID`	32-bit	0
`AFU_REVISION_ID`	12-bit	0

PORT AFU ID

Every target configuration has a unique PORT_AFU_ID:

TARGET	PORT_AFU_ID
8x10G	901DD697-CA79-4B05-B843-8138CEFA2846
4x25G	F3C99413-5081-4AAD-BCED-07EB84A6D0BB
2x2x25G	A5D72A3C-C8B0-4939-912C-F715E5DC10CA

The build process combines your afu.qsf file with a top level chip.qsf that includes external memory interfaces, MACs, and the encrypted CCI-P and management blocks.

To compile the hello_afu targeting the 2x2x25, execute the following in the top directory:

$ cd $N3000_EXAMPLE_ROOT/hello_afu
$ make 2x2x25G GUI=1 INCLUDE_DIAGNOSTICS=0 INCLUDE_MEMORY=0 \ 
PAC_VER_MAJOR=3 PAC_VER_MINOR=5 PAC_VER_PATCH=6 \ 
REVISION_ID=12345678 INCLUDE_AFU_PCIE1=0

This brings up the Intel^® Quartus^® Prime GUI. Click the Start Compilation play button.

Note: The Launch IP Upgrade Tool button appears. You can safely ignore this warning.

When your compile is complete, do not close Intel^® Quartus^® Prime, so that you can continue with the next steps.

4.4. Check Timing

Verify your compiled design meets timing and power requirements by performing the following steps:

Go to the "Tasks" pane and select “Compilation Report”.

Figure 24. Compilation Report
Under Timing Analyzer, verify no failing timing paths.

Figure 25. Timing Analyzer

You can safely ignore the "Unconstrained Paths" report in this release.
Select Power Analyzer and check that “Total Thermal Power Dissipation” is within the thermal characteristics of your server air flow.
Check the Thermal Specifications section of the Intel^® FPGA PAC N3000 Data Sheet.
The power results shown in the power analyzer are based on the worst case FPGA junction temperature of 100° C.
Figure 26. Power Analyzer Summary
Select Project > Generate Early Power Estimator File to perform additional power analysis in the Intel^® Arria^® 10 Early Power Estimator by clicking on this download .
For more information, refer to the Early Power Estimator for Intel^® Arria^® 10 FPGAs User Guide.

The Early Power Estimator (EPE) file can take a few minutes to generate. The default EPE file location is /prj/pac_baseline/chip_early_pwr.csv. This .csv file can be imported into the Early Power Estimator for detailed analysis of power consumption of your design.
Figure 27. Generate Early Power Estimator File

4.5. Loading Your AFU Into the Intel FPGA PAC N3000

Once your design has been compiled using the make process, a new directory is created with all build report files and FPGA programming files. To see these files, do the following after successfully running make:

Note: Must be in the same directory where make was invoked.

$ cd prj/pac_baseline/build/
$ $ ls -1
chip.asm.rpt
chip.done
chip.fit.finalize.rpt
chip.fit.place.rpt
chip.fit.plan.rpt
chip.fit.route.rpt
chip.fit.rpt
chip.fit.summary
chip.flow.rpt
chip_out.sof.rpt
chip.pin
chip.pow.rpt
chip.pow.summary
chip.sld
chip.sta.rpt
chip.sta.summary
chip.syn.rpt
chip.syn.smsg
chip.syn.summary
pac-n3000.map
pac-n3000-secure-update-raw.bin
pac-n3000.sof

The file pac-n3000-secure-update-raw.bin is a binary file formatted to be loaded into the Intel^® FPGA PAC N3000 FPGA flash. Before this file can be loaded into the flash, it must have the prepended authentication blocks generated by PACSign added to the binary file prior to loading using the OPAE tool fpgasupdate. The following instructions guide you in creating an image file with the proper authentication blocks for an Intel^® FPGA PAC N3000 that has not had the root entry hash programmed. Typically, when developing an AFU in a lab environment, do not program the root entry hash until the AFU is production ready.

For more information, refer to the Security User Guide: Intel FPGA Programmable Acceleration Card N3000 .

Before running PACSign, ensure you have the following environment setting:

export PYTHONPATH=/usr/local/lib/python3.6/site-packages/

Create the image and load using fpgasupdate.

$ PACSign SR -t UPDATE -H openssl_manager  \
-i pac-n3000-secure-update-raw.bin  -o unsigned_PAC_N3000_RSU.bin 
No root key specified.  Generate unsigned bitstream? Y = yes, N = no: y
No CSK specified.  Generate unsigned bitstream? Y = yes, N = no: y

By responding with 'y', you are creating an unsigned binary file that can be loaded into a Intel^® FPGA PAC N3000 board that has not had the root key hash loaded into flash.

Perform the fpgasupdate write process.
```
$ sudo fpgasupdate unsigned_PAC_N3000_RSU.bin <PCIe B.D.f>
```
Note: The fpgasupdate write process take approximately 40 minutes to complete.
Once fpgasupdate completes, perform a remote system update to load the new FPGA image, then verify the expected FPGA is loaded with OPAE tools fpgainfo fme and fpgainfo port.
```
$ sudo rsu fpga <PCIe B.D.f>
$ sudo fpgainfo fme 
$ sudo fpgainfo port
```

5. Capturing Signals in AFU with Signal Tap

This section is a short guide on adding remote Signal Tap instances to an AFU for in system debugging. You can follow these steps in order of execution to create an instrumented AFU. The hello_afu project is used in this guide as the target AFU to be instrumented.

You need a basic understanding of Signal Tap. Please see the "Signal Tap Logic Analyzer: Introduction & Getting Started" Web Based Training for more information.

You can run the Signal Tap GUI on a remote laptop or workstation connected through the network to a server with the Intel^® FPGA PAC N3000 with a Signal Tap instrumented FPGA image or you can have a Signal Tap GUI running locally on the server with the Intel^® FPGA PAC N3000. See diagram below illustrating Remote and local debugging scenarios.

Figure 28. Remote Debugging

Figure 29. Local Debugging

Now that you have compiled the hello_afu design, you can proceed as follows for adding Signal Tap to the design.

Familiarize yourself with the project hierarchy. If your Project Navigator is not already in view in your main Intel^® Quartus^® Prime window, then in the Intel^® Quartus^® Prime window select View > Project Navigator. Detach the Project Navigator pane to expand its view. Expand the pac_top instance. Now expand the inst_green_bs instance. Next expand inst_ccip_std_afu. Your Project Navigator display should look as shown below:
For this learning tutorial, the CCI-P interface and scratch register is instrumented. Select the afu instance in Project Navigator and right click, then select Locate Node and finally, Locate in Design File. See screen shot below:
This brings up the hello_afu.sv SystemVerilog code. Having the code available for review while defining the signals to be instrumented is highly useful.
Open the Signal Tap tool to create a *.stp file defining the signals to be instrumented. See example screen shot below on how to open the Signal Tap tool:
1. The Signal Tap GUI appears as shown below:
2. You can now set up a Signal Tap instance to instrument a portion of the design for observability.
  In this example, the Signal Tap instance is renamed to hello_afu to indicate its use to instrument the hello_afu module. To rename the auto_signaltap_0 instance, right click and select Rename Instance:
For the hello_afu Signal Tap instance define the clock used to sample the signals to be instrumented. For accurate results, the instrumented signals must be in the domain of this clock. To select the clock, select the … button under “Signal Configuration”:
1. This brings up the Node Finder tool. Find the “Look in:” listing go to the right and click … to bring up the “Select Hierarchy Level” viewer. See screen shot:
2. In the Select Hierarchy Level viewer expand pac_top, inst_green_bs, inst_ccip_std_afu and select afu and click OK. See screen shot:
3. From reviewing the hello_afu.sv code, notice all signals are synchronous to signal clk. In the Node Finder window, type in *clk* in the “Named” blank and click Search. Expand each instance selection in the “Matching Nodes” pane. Then select clk and the > to make this the signal used for clocking the Signal Tap instance. See screen shot:
  1. Press OK
  2. The Signal Configuration dock on the far right should look as shown below:
You can increase the depth of the samples captured by increasing Sample depth. For this example, the depth is increased to 1 K. Please keep in mind, increased depth means more FPGA resources used for the Signal Tap instance.
Select the signals to be added to the hello_afu Signal Tap instance by double clicking the Double-click to add nodes area of the “Setup” dock. This brings up the Node Finder tool. Repeat the steps above to set the “Look in:” box to narrow the search to just the hello_afu instance. You want to see the CCI-P interface signals and scratch pad register. Enter cp2af_sRxPort* in the "Named" entry field and click Search. Click the >> to select all signals to give you total visibility of the ccip RX bus input to the hello_afu. See screen shot below:
Now change “Named:” to scratch* and click Search. Expand Matching Nodes until the scratch_reg is displayed. Select scratch_reg and then >> to add the scratch register signals to the “Nodes Found” list. See screen shot:
Now enter af2cp_sTxPort* in the "Named" entry field and click Search to instrument the output CCI-P bus signals. Expand the instance names in “Matching Nodes". Do not select names ending in ~reg0. Your display should be as shown below:
Click Insert and Close. Your display should be as shown below:
Select File > Save As and save the newly created Signal Tap Logic Analyzer file as hello_afu.stp in the current directory.
When asked: “Do you want to enable Signal Tap File hello_afu.stp for the current project?” Click Yes.
Close the Signal Tap GUI.
Re-Run a full compilation to create a new FPGA implementation with hello_afu Signal Tap instrumentation included.
Once the build completes, the newly created pac-n3000-secure-update-unsigned.bin with Signal Tap instrumentation can be loaded into FPGA Flash for storage and use as the user image loaded into the Intel^® Arria^® 10 GT FPGA.

Loading FPGA Image

Change directories to the build directory where pac-n3000-secure-update-unsigned.bin was created by make and create an unsigned FPGA image using PACSign. Please note, this step assumes your target Intel^® FPGA PAC N3000 board has not had the root key programmed on the board.
```
$ cd prj/pac_baseline/build
```
Before running PACSign, ensure you have the following environment setting:
```
export PYTHONPATH=/usr/local/lib/python3.6/site-packages/
```

Create the image:

$ PACSign SR -t UPDATE -H openssl_manager  \
-i pac-n3000-secure-update-raw.bin  -o hello_afu_unsigned.bin
No root key specified.  Generate unsigned bitstream? Y = yes, N = no: y
No CSK specified.  Generate unsigned bitstream? Y = yes, N = no: y

By responding with 'y', you are creating an unsigned binary file that can be loaded into a Intel^® FPGA PAC N3000 board that has not had the root key hash loaded into flash.

Flash your Intel^® FPGA PAC N3000 image with this new file.
Note: This command must be done as sudo or root.
```
$ sudo fpgasupdate hello_afu_unsigned.bin 3e:00.0
```
Note: Your PCIe b:d.f value can be different from 3e:00.0 used above.
This command takes about 40 minutes to complete.
Perform a remote system update to load the new FPGA image using your PCIe B.D.f.
```
$ sudo rsu fpga 3e:00.0
```

Verify the hello_afu_stp FPGA image is loaded with the fpgainfo port command showing the Accelerator Id as 850adcc2-6ceb-4b22-9722-d43375b61c66.

# fpgainfo port
Board Management Controller, MAX10 NIOS FW version D.2.0.19
Board Management Controller, MAX10 Build version D.2.0.6
//****** PORT ******//
Object Id : 0xED00000
PCIe s:b:d.f : 0000:3e:00.0
Device Id : 0x0b30
Numa Node : 1
Ports Num : 01
Bitstream Id : 0x21000410030509
Bitstream Version : 0.2.3
Pr Interface Id : a5d72a3c-c8b0-4939-912c-f715e5dc10ca
Accelerator Id : 850adcc2-6ceb-4b22-9722-d43375b61c66

Set Up Connections

Set up a network connection between the instrumented FPGA AFU and your Signal Tap GUI. You can set the network connection based on whether you are using a remote or local debugging configuration. See diagram below illustrating the connection steps:
Figure 30. Remote Debugging

Figure 31. Local Debugging
1. To enable your host system for remote Signal Tap, use the OPAE tool mmlink. This tool is included with the Intel^® Acceleration Stack for the N3000. Open a TCP port to accept incoming connection requests from remote debug hosts.
```
# mmlink -P 3333 –B 0xb2 Note, -B is the bus number of the target N3000 to connect
 ------- Command line Input START ---- 
 Segment : -1
 Bus : 02
 Device : -1
 Function : -1
 Socket-id : -1
 Port : 3333
 IP address : 0.0.0.0
 ------- Command line Input END ---- 
PORT Resource found.
Remote STP : Assert Reset
Remote STP : De-Assert Reset
Read signature value 53797343 to hw
Read version value 1 to hw
Read write fifo capacity value 32 to hw
m_listen: 4
listening on ip: 0.0.0.0; port: 3333
```
2. You can debug remotely from a remote machine connected to your PAC host or you can debug on the local PAC host.
  1. If debugging on a remote host, make sure Intel^® Quartus^® Prime Pro Edition version 19.2 is installed and the directory $N3000_EXAMPLE_ROOT/hello_afu/hw/pac/remote_debug is copied to the remote host.
  2. Use System Console on the remote host to connect to the debug target host IP and TCP port using the following command:
```
$ cd Remote host directory with $N3000_EXAMPLE_ROOT/hello_afu/hw/pac/\
remote_debug
$ system-console --rc_script=mmlink_setup_profiled.tcl \ 
remote_debug.sof <IP Address of target PAC host> 3333 
```
    Note: You must have the System Console executable binary added to your PATH variable. The System Console executable binary is in the Intel^® Quartus^® Prime installation directory. An example of how to update your PATH variable is the following:
```
$PATH:/home/<user name>/inteldevstack/intelFPGA_pro/quartus/sopc_builder/bin
```
  3. If debugging on local host, start System Console on the local host as shown below:
```
$ # cd $N3000_EXAMPLE_ROOT/hello_afu/hw\
/pac/remote_debug 
$ system-console --rc_script=mmlink_setup_profiled.tcl \
remote_debug.sof localhost 3333 
```
    Whether local or remote, the Intel^® Quartus^® Prime tool System Console starts a new GUI and runs the mmlink_setup_profiled.tcl setup script as shown below:
    
    The script takes 1-2 minutes to complete. When the “Remote system ready” message is displayed Signal Tap may be connected for debugging.
3. Open Intel^® Quartus^® Prime GUI on the machine that is performing the debugging (i.e. local or remote host). In Intel^® Quartus^® Prime GUI, select File – Open and navigate to the hello_afu.stp file created in previous steps, select and open this file. The Signal Tap GUI comes up as shown below:
  1. In the Signal Tap GUI top right corner, “Hardware” pull down, where it says “Please Select”, click on the up/down arrows to bring up the hardware selection – select the choice that has System Console …. as shown below:
    
    The instance manager should show "Ready to acquire".
  2. Review the hello_afu.sv code and notice the following line:
  3. Enable the Signal Tap instance to capture data when cp2af_sRxPort.c0.mmioWrValid.
    1. Select the cp2af_sRxPort.co.mmioWrValid signal name, then right click the “Trigger Conditions” value and set this to 1 as shown below:
  4. Enable the hello_afu Signal Tap instance by entering F5.
To create host transactions that can cause the hello_afu STP interface to activate, the Intel^® FPGA PAC N3000 host application needs to be modified to add a shared connection to the FPGA. This shared connection allows the Signal Tap and host communication to be shared through the PCIe* bus.
1. Edit $N3000_EXAMPLE_ROOT/hello_afu/hw/afu/sw/hello_afu.c to change the following line from:
```
res = fpgaOpen(afc_token, &afc_h, 0);
```
  To;
```
res = fpgaOpen(afc_token, &afc_h, FPGA_OPEN_SHARED);
```
2. Save hello_afu.c and build the code
```
$ make
```
3. Run the hello_afu host code as root or sudo as shown below:
```
# ./hello_afu 
Running Test
AFU DFH REG = 1000010000000000
AFU ID LO = 9722d43375b61c66
AFU ID HI = 850adcc26ceb4b22
AFU NEXT = 00000000
AFU RESERVED = 00000000
Reading Scratch Register (Byte Offset=00000080) = 00000000
MMIO Write to Scratch Register (Byte Offset=00000080) = 123456789abcdef
Reading Scratch Register (Byte Offset=00000080) = 123456789abcdef
Setting Scratch Register (Byte Offset=00000080) = 00000000
Reading Scratch Register (Byte Offset=00000080) = 00000000
Done Running Test
```
Your Signal Tap instance captures the write transactions as shown below:

To exit from the debug session, do the following

Close Signal Tap instance.
In System Console, select File > Exit.
In the debug target host shell, terminate the mmlink with a <Ctrl-c> key sequence.
If you want to debug another AFU, you must first terminate the active mmlink process.
Before loading a new AFU, be sure to terminate any OPAE host application code that has the currently loaded AFU open.

Troubleshooting Remote Debug Connections

If you get a Failed to connect message after invoking System Console:

Check if a firewall is blocking your port. If so, unblock your port by running the following:
```
firewall-cmd --add-port=3333/tcp --permanent
```
Consider adding port tunneling. Do this when the debug target host is behind a firewall and your remote debug host is not.

On the debug target host, run mmlink as before. Note that mmlink provides an option to specify a port number. Port 3333 is the default.

Refer to the following:

$ mmlink -- port=3333

Setup port tunneling on the remote debug host. This example shows how to do so on a Windows remote debug host using PuTTY.

Use a PuTTY configuration screen as shown in the SSH Tunneling with PuTTY figure. For <SDP>, enter the name of the debug target host. This forwards the local port on your Windows host 4444 to port 3333 on the debug target host.

Figure 32. SSH Tunneling with PuTTY

Then, Click Session, specify the name of the debug target host, click Save, and then Open. Login to the debug target host. This is your tunneling session.

Figure 33. Save and Open the Tunneling SessionThis figure specifies local host and the port 4444.

Once the tunneling session is setup this forwarding is complete. Open a Windows Command Window and issue the system-console command as shown in the "Save and Open the Tunneling Session" figure.

Run the System Console with Port Forwarding command:

$ system-console --rc_script=mmlink_setup_profiled.tcl remote_debug.sof localhost 4444

As before, the Intel^® Quartus^® Prime System Console comes up. Wait for the Remote system ready message on the tcl console of the System Console.

6. Document Revision History for the Accelerator Funtional Unit Developer Guide for Intel FPGA Programmable Acceleration Card N3000

Document Version	Intel Acceleration Stack Version	Changes
2019.12.06	1.1	Initial Release

BBS_regs_mm_wrap Access Behavior